Moving picture encoding system, moving picture encoding method, moving picture encoding program, moving picture decoding system, moving picture decoding method, moving picture decoding program, moving picture reencoding system, moving picture reencoding method, moving picture reencoding program

ABSTRACT

A first super-resolution enlarger (103) works on moving pictures input with a standard resolution, implementing a process for a super-resolution enlargement including information on frequency components in the spatial and temporal directions that has been potentially contained in the input moving pictures but unable to express to a sufficient degree by the standard resolution, and provides super-resolution enlarged signals, which are returned to the standard resolution at a first resolution converter (104). The super-resolution enlarged signals as returned to the standard resolution are encoded at a second encoder (107). A first encoder (102) encodes moving pictures input with the standard resolution, and a multiplexer (109) multiplexes a sequence of encoded bits from a first encoder (102), a sequence of encoded bits from the second encoder (107), and the like. The second encoder (107) employs local decoded signals in the first encoder (102) or processed signals thereof, as reference signals.

This is a Divisional Application of U.S. patent application Ser. No.12/995,039, filed Nov. 29, 2010, a National Phase Application filedunder 35 U.S.C. 371 as a national stage of PCT/JP2009/059801, filed May28, 2009, an application claiming the benefit from Japanese ApplicationNo. 2008-142433, filed May 30, 2008, and claiming benefit from JapaneseApplication No. 2009-123960, filed May 22, 2009, the content of each ofwhich is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present invention relates to a moving picture encoding system, amoving picture encoding method, a moving picture encoding program, amoving picture decoding system, a moving picture decoding method, amoving picture decoding program, a moving picture reencoding system, amoving picture reencoding method, and a moving picture reencodingprogram adapted to work on moving picture sequences to implementprocesses such as those for super-resolution enlargement.

BACKGROUND ART

As conventional techniques to attain a spatial resolution scalability ofpictures, there have been those implementing, in a hierarchical encodingsystem with two layers being a base layer and an enhancement layer, forinstance, processing signals of pictures input with the same spatialresolution as the enhancement layer, for a decimation into a spatialresolution of the base layer, followed by an encoding at the base layer,making a prediction using a correlation between those signals decodedalong with the base layer encoding and spatially interpolated as signalshaving the same spatial resolution as the enhancement layer and thosesignals of pictures input with the same spatial resolution as theenhancement layer, encoding signals of errors in the prediction, andhaving a combination of encoded bit streams obtained there and bitstreams obtained by the base layer encoding, multiplexed and transmittedto a decoding system, the multiplexed combination of encoded bit streamsbeing decoded in reverse at the decoding system. (Refer to PatentLiterature 1.)

CITATION LIST Patent Literature

PTL 1: Japanese Patent Application Laying-Open Publication No. 7-162870

SUMMARY OF THE INVENTION Technical Problem

By the way, there are techniques for moving picture encoding servicessuch as represented by MPEG-1, -2, and -4/AVC/SVC specified tostandardize under the ISO/IEC SC29WG11, including use of decodedpictures obtained by a local decoding as reference pictures forestimation of motions to create data on motion vectors, and use of dataon motion vectors created by compensation of motions, to create suchpredictive pictures as having nearest image qualities to target picturesas encoding targets, followed by determining difference data betweenpredictive pictures and target pictures, and implementing on thedifference data a process for a prescribed encoding to attain a highefficient encoding making use of high correlations in the temporaldirection. It is thus possible to have predictive pictures improved inimage quality, and enhanced in correlation to target pictures, affordingto expect more enhanced encoding efficiencies. However, failures toallot a sufficient code rate would cause deficiencies such as deficientprecision of data on motion vectors or deficient attainment of thequality of images of locally decoded reference pictures, degrading imagequalities of predictive pictures, resulting in reduced encodingefficiencies.

Moreover, there are techniques for hierarchical encoding services onmoving pictures represented by among others the above-noted PatentLiterature 1 and MPEG-4 SVC, which also include predictive pictures orpredictive blocks created from reference pictures of a layer lower thanthe layer in which a current encoding is made, for use in combinationwith target pictures or target blocks of the current layer to make aninter-layer prediction in between, aiming at still enhanced encodingefficiencies making use of high correlations between different spatialresolutions. However, like techniques making use of high correlations inthe temporal direction, also those making use of high correlationsbetween layers are subject to the problem that failures to allot asufficient code rate would degrade image qualities of predictivepictures or predictive blocks, resulting in reduced encodingefficiencies.

Further, they involve implementing a process for a prescribed resolutionconversion on moving pictures input as encoding targets with a spatialresolution (referred herein to as a base or standard resolution),creating moving pictures with a resolution (referred herein to as a lowresolution) lower than the standard resolution of the input movingpictures, to make a hierarchical encoding on moving pictures between twoor more layers making use of high correlations between layers. However,in such the hierarchical encoding, even if the encoding processimplemented was reversible, input moving pictures would undergo a bandlimitation to information on their spatial frequency components in thespatial direction that can be represented by the standard resolution, inaddition to the implementation of a hierarchical encoding including aprocess to be implemented on the input moving pictures for such anencoding that involves decoded pictures obtained through a localdecoding in course of hierarchical encoding, or by decoding bit streamsencoded by way of hierarchical encoding, with image qualities nearest tothose of input moving pictures under given encoding conditions, thusaffording to make use of correlations between associated layers toestimate new information on spatial frequency components, while failingto encode and transmit more information on spatial frequency componentsthan the information on spatial frequency components that can berepresented by the standard resolution, as a problem.

It therefore is an object of the present invention to provide a movingpicture encoding system, a moving picture encoding method, and a movingpicture encoding program adapted to encode and transmit more informationon spatial frequency components than information on spatial frequencycomponents that can be represented by a resolution of input movingpictures. It also is an object of the present invention to provide amoving picture decoding system, a moving picture decoding method, and amoving picture decoding program adapted to acquire and decode sequencesof encoded bits created by a moving picture encoding system, a movingpicture encoding method, or a moving picture encoding program accordingto the present invention. It also is an object of the present inventionto provide a moving picture reencoding system, a moving picturereencoding method, and a moving picture reencoding program adapted for adecoding followed by a re-encoding and a transmission.

Solution to Problem

To solve the problems described, according to the present invention,there is a moving picture encoding system comprising, as illustrated inFIG. 9 for instance, a first encoder configured to work on a subsequenceof a sequence of moving pictures with a standard resolution to implementa first combination of processes for an encoding and a decoding tocreate a first sequence of encoded bits and a set of decoded pictureswith the standard resolution, a first super-resolution enlargerconfigured to work on the subsequence of the sequence of moving pictureswith the standard resolution to implement a process for a firstsuper-resolution enlargement to create a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a first resolution converter configured to work on the set ofsuper-resolution enlarged pictures to implement a process for a firstresolution conversion to create a set of super-resolution enlarged andconverted pictures with the standard resolution, and a second encoderconfigured to have the set of super-resolution enlarged and convertedpictures from the first resolution converter as a set of encoding targetpictures to implement a second combination of processes for a predictionand an encoding to create a second sequence of encoded bits.

Here, as illustrated in FIG. 7 for instance, it may well have the secondencoder configured to have the set of decoded pictures from the firstencoder as a set of reference pictures to implement the secondcombination of processes for the prediction and the encoding to createthe second sequence of encoded bits.

Further, as illustrated in FIGS. 1 and 8 for instance, it may wellcomprise a second super-resolution enlarger configured to acquire theset of decoded pictures with the standard resolution from the firstencoder to implement thereon a process for a second super-resolutionenlargement to create a set of super-resolution enlarged decodedpictures with a resolution higher than the standard resolution, and asecond resolution converter configured to work on the set ofsuper-resolution enlarged decoded pictures to implement a process for asecond resolution conversion to create a set of super-resolutionenlarged and converted decoded pictures with the standard resolution,the second encoder being configured to have the set of super-resolutionenlarged and converted decoded pictures from the second resolutionconverter as a set of reference pictures to implement the secondcombination of processes for the prediction and the encoding to createthe second sequence of encoded bits.

Further, as illustrated at (a) in FIG. 4 for instance, it may well havethe first super-resolution enlarger comprising a positioner configuredto work in a processing for super-resolution enlargement, on acombination of a base picture constituting a base therein and one ormore observation pictures to be based on, to make a positioning to pixelpositions of a desirable high resolution, for an interval-unequalsampling to create nonhomogeneous high resolution pictures, aninterpolator configured to work on nonhomogeneous high resolutionpictures created at the positioner, to implement a process for aprescribed nonhomogeneous interpolation to create interpolated pictureswith a desirable high resolution, an estimated picture creatorconfigured to acquire interpolated pictures created at the interpolator,to implement thereon a process for a prescribed reconstruction to createestimated pictures with a desirable resolution, and a repetitiondeterminer configured to acquire a nonhomogeneous high resolutionpicture from the positioner and a nonhomogeneous estimated picture fromthe estimated picture creator, for employment of the acquired picturesto follow a prescribed determination method to determine whether or nota repetition of the processing for super-resolution enlargement isnecessary, and work depending on a result thereof, to operate as therepetition is necessary, to provide the interpolator with informationfor a control to continue the processing, and operate as the repetitionis unnecessary, to provide the estimated picture creator withinformation for a control to output a set of estimated pictures with ahigh resolution after super-resolution enlargement.

Further, as illustrated in FIG. 1 or such for instance, it may wellcomprise a multiplexer configured to operate complying with a prescribedsyntax structure to multiplex the first sequence of encoded bits fromthe first encoder, the second sequence of encoded bits from the secondencoder, and information on encoding parameters of respective types usedin the encoding at the first encoder and the encoding at the secondencoder.

Further, as illustrated in FIG. 10 for instance, it may well comprise afirst accumulation buffer configured to work in a course before thefirst encoder and the first super-resolution enlarger, to accumulatesubsequences of the sequence of moving pictures with the standardresolution, a second accumulation buffer configured to work in a coursebetween the first encoder and the second encoder, to accumulate sets ofdecoded pictures from the first encoder, an accumulation controllerconfigured to work for detections of buffer accumulation amounts at thefirst accumulation buffer, the second accumulation buffer, the firstencoder, and the second encoder, to control the buffer accumulationamounts, and a code rate controller configured to work on bases of thedetections of buffer accumulation amounts at the accumulationcontroller, to control code rates at the first encoder and the secondencoder.

Further, according to the present invention, there is a moving pictureencoding system comprising, as illustrated in FIG. 13 for instance, afirst encoder configured to work on a sequence of moving pictures with astandard resolution to implement a first combination of processes for anencoding and a decoding to create a first sequence of encoded bits and aset of decoded pictures with the standard resolution, a firstsuper-resolution enlarger configured to work on the sequence of movingpictures with the standard resolution to implement a process for a firstsuper-resolution enlargement to create a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a second super-resolution enlarger configured to acquire the set ofdecoded pictures from the first encoder to implement thereon a processfor a second super-resolution enlargement to create a set ofsuper-resolution enlarged decoded pictures with a resolution higher thanthe standard resolution, a third resolution converter configured toacquire the set of decoded pictures from the first encoder to implementthereon a process for a third resolution conversion to create a set ofresolution converted enlarged decoded pictures with a resolution higherthan the standard resolution, and a third encoder configured to have theset of super-resolution enlarged pictures from the firstsuper-resolution enlarger as a set of encoding target pictures,employing the set of super-resolution enlarged decoded pictures from thesecond super-resolution enlarger and the set of resolution convertedenlarged decoded pictures from the third resolution converter as sets ofreference pictures, to implement thereon a third combination ofprocesses for a prediction and an encoding to create a third sequence ofencoded bits.

Further, according to the present invention, there is a moving pictureencoding system comprising, as illustrated in FIG. 16 for instance, afirst encoder configured to work on a sequence of moving pictures with astandard resolution to implement a first combination of processes for anencoding and a decoding to create a first sequence of encoded bits and aset of decoded pictures with the standard resolution, a firstsuper-resolution enlarger configured to work on the sequence of movingpictures with the standard resolution to implement a process for a firstsuper-resolution enlargement to create a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a third resolution converter configured to acquire the set of decodedpictures from the first encoder to implement thereon a process for athird resolution conversion to create a set of resolution convertedenlarged decoded pictures with a resolution higher than the standardresolution, and a third encoder configured to have the set ofsuper-resolution enlarged pictures from the first super-resolutionenlarger as a set of encoding target pictures, employing the set ofresolution converted enlarged decoded pictures from the third resolutionconverter as a set of reference pictures, to implement thereon a thirdcombination of processes for a prediction and an encoding to create athird sequence of encoded bits.

Further, according to the present invention, there is a moving pictureencoding system comprising, as illustrated in FIG. 17 for instance, afirst encoder configured to work on a sequence of moving pictures with astandard resolution to implement a first combination of processes for anencoding and a decoding to create a first sequence of encoded bits and aset of decoded pictures with the standard resolution, a firstsuper-resolution enlarger configured to work on the sequence of movingpictures with the standard resolution to implement a process for a firstsuper-resolution enlargement to create a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a second super-resolution enlarger configured to acquire the set ofdecoded pictures from the first encoder to implement thereon a processfor a second super-resolution enlargement to create a set ofsuper-resolution enlarged decoded pictures with a resolution higher thanthe standard resolution, and a third encoder configured to have the setof super-resolution enlarged pictures from the first super-resolutionenlarger as a set of encoding target pictures, employing the set ofsuper-resolution enlarged decoded pictures from the secondsuper-resolution enlarger as a set of reference pictures, to implementthereon a third combination of processes for a prediction and anencoding to create a third sequence of encoded bits.

Further, according to the present invention, there is a moving pictureencoding system comprising, as illustrated in FIG. 19 for instance, afirst encoder configured to work on a sequence of moving pictures with astandard resolution to implement a first combination of processes for anencoding and a decoding to create a first sequence of encoded bits and aset of decoded pictures with the standard resolution, a firstsuper-resolution enlarger configured to work on the sequence of movingpictures with the standard resolution to implement a process for a firstsuper-resolution enlargement to create a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a first resolution converter configured to work on the set ofsuper-resolution enlarged pictures to implement a process for a firstresolution conversion to create a set of super-resolution enlarged andconverted pictures with the standard resolution, a secondsuper-resolution enlarger configured to acquire the set of decodedpictures from the first encoder to implement thereon a process for asecond super-resolution enlargement to create a set of super-resolutionenlarged decoded pictures with a resolution higher than the standardresolution, a second resolution converter configured to work on the setof super-resolution enlarged decoded pictures from the secondsuper-resolution enlarger to implement a process for a second resolutionconversion to create a set of super-resolution enlarged and converteddecoded pictures with the standard resolution, a second encoderconfigured to have the set of super-resolution enlarged and convertedpictures from the first resolution converter as a set of encoding targetpictures, employing the set of decoded pictures from the first encoderand the set of super-resolution enlarged and converted decoded pictureswith the standard resolution from the second resolution converter assets of reference pictures, to implement thereon a second combination ofprocesses for a prediction and an encoding to create a second sequenceof encoded bits, a third resolution converter configured to acquire theset of decoded pictures from the first encoder to implement thereon aprocess for a third resolution conversion to create a set of resolutionconverted enlarged decoded pictures with a resolution higher than thestandard resolution, and a third encoder configured to have the set ofsuper-resolution enlarged pictures from the first super-resolutionenlarger as a set of encoding target pictures, employing the set ofsuper-resolution enlarged decoded pictures from the secondsuper-resolution enlarger and the set of resolution converted enlargeddecoded pictures from the third resolution converter as sets ofreference pictures, to implement thereon a third combination ofprocesses for a prediction and an encoding to create a third sequence ofencoded bits.

Further, according to the present invention, there is a moving pictureencoding method comprising a step of implementing a first combination ofprocesses for an encoding and a decoding on a sequence of movingpictures with a standard resolution, creating a first sequence ofencoded bits and a set of decoded pictures with the standard resolution,a step of implementing a process for a first super-resolutionenlargement on the sequence of moving pictures with the standardresolution, creating a set of super-resolution enlarged pictures with aresolution higher than the standard resolution, a step of implementing aprocess for a first resolution conversion on the set of super-resolutionenlarged pictures, creating a set of super-resolution enlarged andconverted pictures with the standard resolution, and a step of havingthe set of super-resolution enlarged and converted pictures as a set ofencoding target pictures, implementing thereon a second combination ofprocesses for a prediction and an encoding, creating a second sequenceof encoded bits.

Further, according to the present invention, there is a moving pictureencoding program configured to have a computer execute a step ofimplementing a first combination of processes for an encoding and adecoding on a sequence of moving pictures with a standard resolution,creating a first sequence of encoded bits and a set of decoded pictureswith the standard resolution, a step of implementing a process for afirst super-resolution enlargement on the sequence of moving pictureswith the standard resolution, creating a set of super-resolutionenlarged pictures with a resolution higher than the standard resolution,a step of implementing a process for a first resolution conversion onthe set of super-resolution enlarged pictures, creating a set ofsuper-resolution enlarged and converted pictures with the standardresolution, and a step of having the set of super-resolution enlargedand converted pictures as a set of encoding target pictures,implementing thereon a second combination of processes for a predictionand an encoding, creating a second sequence of encoded bits.

Further, according to the present invention, there is a moving picturedecoding system comprising, as illustrated in FIG. 26 for instance, ademultiplexer configured to work on a sequence of input encoded bits toimplement a process for a prescribed demultiplexing to output sequencesof encoded bits with a standard resolution, a first decoder configuredto acquire a sequence of encoded bits obtained with the standardresolution at the demultiplexer to implement thereon a process for aprescribed decoding to create a sequence of decoded pictures with thestandard resolution, a first super-resolution enlarger configured toacquire the sequence of decoded pictures created with the standardresolution at the first decoder to implement thereon a process for aprescribed super-resolution enlargement to create a sequence ofsuper-resolution enlarged decoded pictures, a first resolution converterconfigured to acquire the sequence of super-resolution enlarged decodedpictures created at the first super-resolution enlarger to implementthereon a process for a prescribed resolution conversion to create asequence of super-resolution decoded pictures with the standardresolution, and a second decoder configured to acquire a sequence ofencoded bits obtained with an extension of the standard resolution atthe demultiplexer, the sequence of decoded pictures created with thestandard resolution at the first decoder, and the sequence ofsuper-resolution decoded pictures created with the standard resolutionat the first resolution converter, to implement thereon processesincluding a prescribed second decoding being a decoding with anextension of the standard resolution, to create a sequence ofsuper-resolution pictures decoded with the standard resolution.

Here, the moving picture decoding system may well comprise, asillustrated in FIG. 23 for instance, a second resolution converterconfigured to acquire the sequence of decoded pictures with the standardresolution from the decoder to implement thereon a process for aprescribed resolution conversion to create a sequence of enlargeddecoded pictures with a high resolution as a resolution higher than thestandard resolution, and a third decoder configured to acquire asequence of encoded bits obtained with an extension of the highresolution at the demultiplexer, the sequence of super-resolutionenlarged decoded pictures created at the super-resolution enlarger, andthe sequence of enlarged decoded pictures with the high resolution fromthe second resolution converter, to implement thereon a combination ofprocesses for a prescribed prediction and a prescribed decoding, tocreate a sequence of super-resolution enlarged pictures as decoded.

Further, according to the present invention, there is a moving picturereencoding system comprising, as illustrated in FIG. 27 for instance, ademultiplexer configured to work on a sequence of input encoded bits toimplement a process for a prescribed demultiplexing to output sequencesof encoded bits with a standard resolution, a decoder configured toacquire a sequence of encoded bits obtained with the standard resolutionat the demultiplexer to implement thereon a process for a prescribeddecoding to create a sequence of decoded pictures with the standardresolution, a first super-resolution enlarger configured to acquire thesequence of decoded pictures created with the standard resolution at thedecoder to implement thereon a process for a prescribed super-resolutionenlargement to create a sequence of super-resolution enlarged decodedpictures, a first resolution converter configured to acquire thesequence of super-resolution enlarged decoded pictures created at thefirst super-resolution enlarger to implement thereon a process for aprescribed resolution conversion to create a sequence ofsuper-resolution decoded pictures with the standard resolution, a seconddecoder configured to acquire a sequence of encoded bits obtained withan extension of the standard resolution at the demultiplexer, thesequence of decoded pictures created with the standard resolution at theabove-noted decoder, and the sequence of super-resolution decodedpictures created with the standard resolution at the first resolutionconverter, to implement thereon a process for a prescribed seconddecoding being a decoding with an extension of the standard resolution,to create a sequence of super-resolution pictures decoded with thestandard resolution, a reencoder configured to acquire from theabove-noted decoder information on coefficients of orthogonal transformin a course of decoding thereof, and from the second decoder informationon coefficients of orthogonal transform in an extension layer of thestandard resolution in a course of decoding thereof, to make a synthesisof respective information on coefficients of orthogonal transform andimplement thereon a process for a prescribed entropy encoding to createa sequence of encoded bits as reencoded, and a multiplexer configured toacquire the sequence of encoded bits reencoded at the reencoder, toimplement thereon a process for a multiplexing complying with aprescribed syntax structure, inserting encoding information inclusive ofidentification information for identification of information on encodingmodes and parameters of respective types used, to create a sequence ofencoded bits as multiplexed.

Here, the moving picture reencoding system may well have the reencoderconfigured to acquire the sequence of decoded pictures with the standardresolution from the above-noted decoder, the sequence ofsuper-resolution decoded pictures with the standard resolution from thefirst resolution converter, and the sequence of super-resolutionpictures with the standard resolution from the second decoder, toimplement thereon a process for a prescribed encoding to create asequence of encoded bits as reencoded.

Further, according to the present invention, there is a moving picturedecoding method comprising a step of implementing a process for aprescribed demultiplexing on a sequence of input encoded bits,outputting sequences of encoded bits with a standard resolution, a stepof acquiring a sequence of encoded bits obtained with the standardresolution through the process for the prescribed demultiplexing,implementing thereon a process for a prescribed decoding, creating asequence of decoded pictures with the standard resolution, a step ofacquiring the sequence of decoded pictures created with the standardresolution through the process for the above-noted prescribed decoding,implementing thereon a process for a prescribed super-resolutionenlargement, creating a sequence of super-resolution enlarged decodedpictures, a step of acquiring the sequence of super-resolution enlargeddecoded pictures created through the process for the prescribedsuper-resolution enlargement, implementing thereon a process for aprescribed resolution conversion, creating a sequence ofsuper-resolution decoded pictures with the standard resolution, and astep of acquiring a sequence of encoded bits obtained with an extensionof the standard resolution through the process for the prescribeddemultiplexing, the sequence of decoded pictures created with thestandard resolution through the process for the above-noted prescribeddecoding, and the sequence of super-resolution decoded pictures createdwith the standard resolution through the process for the prescribedresolution conversion, implementing thereon processes including aprescribed second decoding being a decoding with an extension of thestandard resolution, creating a sequence of super-resolution picturesdecoded with the standard resolution.

Here, the moving picture decoding method may well comprise a step ofacquiring the sequence of decoded pictures created with the standardresolution through the process for the above-noted prescribed decoding,implementing thereon a process for a prescribed resolution conversion,creating a sequence of enlarged decoded pictures with a high resolutionas a resolution higher than the standard resolution, and a step ofacquiring a sequence of encoded bits obtained with an extension of thehigh resolution through the process for the prescribed demultiplexing,the sequence of super-resolution enlarged decoded pictures createdthrough the process for the prescribed super-resolution enlargement, andthe sequence of enlarged decoded pictures created with the highresolution through the process for the prescribed resolution conversion,implementing thereon a combination of processes for a prescribedprediction and a prescribed decoding, creating a sequence ofsuper-resolution enlarged pictures as decoded.

Further, according to the present invention, there is a moving picturereencoding method comprising a step of implementing a process for aprescribed demultiplexing on a sequence of input encoded bits, a step ofacquiring a sequence of encoded bits obtained with a standard resolutionthrough the process for the prescribed demultiplexing, implementingthereon a process for a prescribed decoding, creating a sequence ofdecoded pictures with the standard resolution, a step of acquiring thesequence of decoded pictures created with the standard resolutionthrough the process for the above-noted prescribed decoding,implementing thereon a process for a prescribed super-resolutionenlargement, creating a sequence of super-resolution enlarged decodedpictures, a step of acquiring the sequence of super-resolution enlargeddecoded pictures created through the process for the prescribedsuper-resolution enlargement, implementing thereon a process for aprescribed resolution conversion, creating a sequence ofsuper-resolution decoded pictures with the standard resolution, a stepof acquiring a sequence of encoded bits obtained with an extension ofthe standard resolution through the process for the prescribeddemultiplexing, the sequence of decoded pictures created with thestandard resolution through the process for the above-noted prescribeddecoding, and the sequence of super-resolution decoded pictures createdwith the standard resolution through the process for the prescribedresolution conversion, implementing thereon a process for a prescribedsecond decoding being a decoding with an extension of the standardresolution, creating a sequence of super-resolution pictures decodedwith the standard resolution, a step of acquiring information oncoefficients of orthogonal transform in a course of decoding in theprocess for the above-noted prescribed decoding, and information oncoefficients of orthogonal transform in an extension layer of thestandard resolution in a course of decoding in the process for theprescribed second decoding being a decoding with an extension of thestandard resolution, making a synthesis of respective information oncoefficients of orthogonal transform, implementing thereon a process fora prescribed entropy encoding, creating a sequence of encoded bits asreencoded, and a step of acquiring the sequence of encoded bits asreencoded, implementing thereon a process for a multiplexing complyingwith a prescribed syntax structure, inserting encoding informationinclusive of identification information for identification ofinformation on encoding modes and parameters of respective types used,creating a sequence of encoded bits as multiplexed.

Further, according to the present invention, there is a moving picturedecoding program configured to have a computer execute a step ofimplementing a process for a prescribed demultiplexing on a sequence ofinput encoded bits, outputting sequences of encoded bits with a standardresolution, a step of acquiring a sequence of encoded bits obtained withthe standard resolution through the process for the prescribeddemultiplexing, implementing thereon a process for a prescribeddecoding, creating a sequence of decoded pictures with the standardresolution, a step of acquiring the sequence of decoded pictures createdwith the standard resolution through the process for the above-notedprescribed decoding, implementing thereon a process for a prescribedsuper-resolution enlargement, creating a sequence of super-resolutionenlarged decoded pictures, a step of acquiring the sequence ofsuper-resolution enlarged decoded pictures created through the processfor the prescribed super-resolution enlargement, implementing thereon aprocess for a prescribed resolution conversion, creating a sequence ofsuper-resolution decoded pictures with the standard resolution, and astep of acquiring a sequence of encoded bits obtained with an extensionof the standard resolution through the process for the prescribeddemultiplexing, the sequence of decoded pictures created with thestandard resolution through the process for the above-noted prescribeddecoding, and the sequence of super-resolution decoded pictures createdwith the standard resolution through the process for the prescribedresolution conversion, implementing thereon processes including aprescribed second decoding being a decoding with an extension of thestandard resolution, creating a sequence of super-resolution picturesdecoded with the standard resolution.

Further, according to the present invention, there is a moving picturereencoding program configured to have a computer execute a step ofimplementing a process for a prescribed demultiplexing on a sequence ofinput encoded bits, a step of acquiring a sequence of encoded bitsobtained with a standard resolution through the process for theprescribed demultiplexing, implementing thereon a process for aprescribed decoding, creating a sequence of decoded pictures with thestandard resolution, a step of acquiring the sequence of decodedpictures created with the standard resolution through the process forthe above-noted prescribed decoding, implementing thereon a process fora prescribed super-resolution enlargement, creating a sequence ofsuper-resolution enlarged decoded pictures, a step of acquiring thesequence of super-resolution enlarged decoded pictures created throughthe process for the prescribed super-resolution enlargement,implementing thereon a process for a prescribed resolution conversion,creating a sequence of super-resolution decoded pictures with thestandard resolution, a step of acquiring a sequence of encoded bitsobtained with an extension of the standard resolution through theprocess for the prescribed demultiplexing, the sequence of decodedpictures created with the standard resolution through the process forthe above-noted prescribed decoding, and the sequence ofsuper-resolution decoded pictures created with the standard resolutionthrough the process for the prescribed resolution conversion,implementing thereon a process for a prescribed second decoding being adecoding with an extension of the standard resolution, creating asequence of super-resolution pictures decoded with the standardresolution, a step of acquiring information on coefficients oforthogonal transform in a course of decoding in the process for theabove-noted prescribed decoding, and information on coefficients oforthogonal transform in an extension layer of the standard resolution ina course of decoding in the process for the prescribed second decodingbeing a decoding with an extension of the standard resolution, making asynthesis of respective information on coefficients of orthogonaltransform, implementing thereon a process for a prescribed entropyencoding, creating a sequence of encoded bits as reencoded, and a stepof acquiring the sequence of encoded bits as reencoded, implementingthereon a process for a multiplexing complying with a prescribed syntaxstructure, inserting encoding information inclusive of identificationinformation for identification of information on encoding modes andparameters of respective types used, creating a sequence of encoded bitsas multiplexed.

Advantageous Effects of the Invention

According to the present invention, there are systems such as a movingpicture encoding system, having a first encoder working for services toencode moving pictures input with a standard resolution, in combinationwith a first super-resolution enlarger working on moving pictures inputwith the standard resolution, implementing a process for asuper-resolution enlargement including information on frequencycomponents in the spatial direction and the temporal direction that hasbeen potentially contained in the input moving pictures but unable toexpress to a sufficient degree by the standard resolution, followed byimplementing processes such as for a prescribed resolution conversion ata first resolution converter, thus permitting a second encoder to makean encoding of moving pictures based on an increased amount ofinformation relative to an information amount of moving pictures inputwith the standard resolution, as a resultant effect. At the secondencoder, as reference signals for the encoding there may be use of,among others, decoded signals as decoded at the first encoder or asadditionally processed through processes such as for a secondsuper-resolution enlargement and a second resolution conversion.

Moreover, according to the present invention, there are systems such asa moving picture encoding system, having a first encoder working forservices to encode moving pictures input with a standard resolution, incombination with a first super-resolution enlarger working on movingpictures input with the standard resolution, implementing a process fora super-resolution enlargement including information on frequencycomponents in the spatial direction and the temporal direction that hasbeen potentially contained in the input moving pictures but unable toexpress to a sufficient degree by the standard resolution, followed byan encoding of thus obtained signals at a third encoder, permitting thethird encoder to make an encoding of moving pictures based on anincreased amount of information relative to an information amount ofmoving pictures input with the standard resolution, as a resultanteffect. At the third encoder, as reference signals for the encodingthere may be use of decoded signals as decoded at the first encoder andprocessed through processes such as for a second super-resolutionenlargement or a third resolution conversion to enhance the resolutionup to the same resolution as first super-resolution enlarged signals.Still more, according to the present invention, there are systems suchas a moving picture decoding system, adapted to input thus encodedmoving pictures to decode, and yet more, according to the presentinvention, there are systems such as a moving picture reencoding system,adapted to input thus encoded moving pictures to decode and reencode.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example of basic configuration of amoving picture encoding system according to a first embodiment of thepresent invention.

FIG. 2 is a flowchart of actions of the moving picture encoding systemaccording to the first embodiment.

FIG. 3 is a diagram of data format illustrating an example of structureof a multiplexed bit sequence created at a multiplexer 109 in the firstembodiment.

FIG. 4 is a combination of a block diagram showing an example ofdetailed configuration of a first super-resolution enlarger 103 in thefirst embodiment, and conceptual diagrams illustrating an example ofprocess for a prescribed super-resolution enlargement to be implementedat the first super-resolution enlarger 103.

FIG. 5 is a flowchart of actions of the first super-resolution enlarger103 in the first embodiment.

FIG. 6 is a combination of a block diagram showing an example ofdetailed structure of a first resolution converter 104 in the firstembodiment, and conceptual diagrams illustrating an example of processfor a prescribed resolution conversion to be implemented at the firstresolution converter 104.

FIG. 7 is a block diagram showing an example of configuration of amoving picture encoding system according to a second embodiment.

FIG. 8 is a block diagram showing another example of configuration ofthe moving picture encoding system according to the second embodiment.

FIG. 9 is a block diagram showing still another example of configurationof the moving picture encoding system according to the secondembodiment.

FIG. 10 is a block diagram showing an example of configuration of amoving picture encoding system according to a third embodiment.

FIG. 11 is a block diagram showing an example of configuration of afirst encoder 403 in the third embodiment.

FIG. 12 is a block diagram showing an example of configuration of afirst decoder 404 in the third embodiment.

FIG. 13 is a block diagram showing an example of configuration of amoving picture encoding system according to a fourth embodiment.

FIG. 14 is a flowchart of exemplary actions of the moving pictureencoding system according to the fourth embodiment.

FIG. 15 is a diagram of data format illustrating an example of structureof a multiplexed bit sequence created at a multiplexer 109 in the fourthembodiment.

FIG. 16 is a block diagram showing an example of configuration of amoving picture encoding system according to a fifth embodiment.

FIG. 17 is a block diagram showing another example of configuration ofthe moving picture encoding system according to the fifth embodiment.

FIG. 18 is a block diagram showing an example of configuration of amoving picture encoding system according to a sixth embodiment.

FIG. 19 is a block diagram showing an example of configuration of amoving picture encoding system according to a seventh embodiment.

FIG. 20 is a flowchart of exemplary actions of the moving pictureencoding system according to the seventh embodiment.

FIG. 21 is a diagram of data format illustrating an example of structureof a multiplexed bit sequence created at a multiplexer 109 in theseventh embodiment.

FIG. 22 is a block diagram showing an example of configuration of amoving picture encoding system according to an eighth embodiment.

FIG. 23 is a block diagram showing an example of configuration of amoving picture decoding system according to a ninth embodiment.

FIG. 24 is a flowchart of exemplary actions of the moving picturedecoding system according to the ninth embodiment.

FIG. 25 is a flowchart showing an example of operational procedure of aprocess for a decoding at a standard resolution at a step S602 in FIG.24.

FIG. 26 is a block diagram showing an example of configuration of amoving picture decoding system according to a tenth embodiment.

FIG. 27 is a block diagram showing an example of configuration of amoving picture reencoding system according to an eleventh embodiment.

FIG. 28 is a flowchart of exemplary actions of the moving picturereencoding system according to the eleventh embodiment.

DESCRIPTION OF EMBODIMENTS

(First Embodiment)

FIG. 1 is a block diagram showing an example of configuration of amoving picture encoding system according to a first embodiment of thepresent invention.

Referring to FIG. 1, the moving picture encoding system according to thefirst embodiment includes a first accumulation buffer 101, a firstencoder 102, a first super-resolution enlarger 103, a first resolutionconverter 104, a second super-resolution enlarger 105, a secondresolution converter 106, a second encoder 107, and a multiplexer 109.

The first accumulation buffer 101 is configured for functions to work onmoving pictures input as an input, to accumulate two or more pictures asnecessary to implement later-described processes for a prescribedpre-super-resolution encoding and a first super-resolution enlargement,and supply the first encoder 102, the first super-resolution enlarger103, and the like with pictures they require.

The first encoder 102 is configured for functions to acquire from thefirst accumulation buffer 101 a set of pictures with a standardresolution required for a process for a prescribed encoding, and work onpictures acquired with the standard resolution to implement the processfor the prescribed encoding (referred herein to as a process for apre-super-resolution encoding), to create a first sequence of encodedbits, to supply the first sequence of encoded bits thus created to themultiplexer 109. The first encoder 102 is configured for functions toimplement a process for a prescribed decoding, to supply decodedpictures to the second super-resolution enlarger 105 and the secondencoder 107. The first encoder 102 may well be configured for functionsto temporarily store decoded pictures. There may be a configuration toimplement the process for the prescribed decoding as a decoding processon the first sequence of encoded bits as created to create decodedpictures. There may be configuration for use of a process for a localdecoding constituting part of the process for the prescribed encoding toemploy results of the local decoding to create pictures decodedtherefrom.

The first super-resolution enlarger 103 is configured for functions toacquire from the first accumulation buffer 101 two or more pictures withthe standard resolution, as necessary for a process for a prescribedsuper-resolution enlargement, and work on pictures acquired with thestandard resolution to implement the process for the prescribedsuper-resolution enlargement, to create super-resolution enlargedpictures with a resolution higher than the standard resolution, andsupply thus created super-resolution enlarged pictures to the firstresolution converter 104. There may be configuration with a function tohave a ratio of enlargement set up upon implementation of the processfor the prescribed super-resolution enlargement as preset information ona prescribed enlargement ratio, or as acquired information onenlargement ratio such as set up by an external user, and work on thebasis of acquired information on enlargement ratio to establish anenlargement ratio to implement the process for the prescribedsuper-resolution enlargement. More preferably, to provide information onthe prescribed enlargement ratio to a decoding system end, there shouldbe use of a configuration to implement an unshown process for aprescribed entropy encoding, to create a bit sequence of information onenlargement ratio, to supply to the multiplexer 109, to transmit to thatend. In this regard, there should be use of similar configuration forthe second super-resolution enlarger 105 also. The firstsuper-resolution enlarger 103 and the second super-resolution enlarger105 have their configurations and actions, of which detailed exampleswill be described with reference to FIG. 4.

The first resolution converter 104 is configured for functions toacquire super-resolution enlarged pictures from the firstsuper-resolution enlarger 103, and work on acquired super-resolutionenlarged pictures to implement a process for a prescribed resolutionconversion to create, from the super-resolution enlarged pictures havinga resolution higher than the standard resolution, a set of pictures (asa set of super-resolution enlarged and converted pictures) such ashaving the standard resolution for instance, to supply to the secondencoder 107.

The second super-resolution enlarger 105 is configured for functions toacquire from the first encoder 102 two or more decoded pictures with thestandard resolution, as necessary for a process for a prescribedsuper-resolution enlargement, and work on pictures acquired with thestandard resolution to implement the process for the prescribedsuper-resolution enlargement, to create super-resolution enlargeddecoded pictures with a resolution higher than the standard resolution,and supply thus created super-resolution enlarged decoded pictures tothe second resolution converter 106. Like the first super-resolutionenlarger 103, there may be configuration with a function to have a ratioof enlargement set up upon implementation of the process for theprescribed super-resolution enlargement as preset information on aprescribed enlargement ratio, or as acquired information on enlargementratio such as set up by an external user, and work on the basis ofacquired information on enlargement ratio to establish an enlargementratio to implement the process for the prescribed super-resolutionenlargement. It is noted that the resolution after enlargement at thesecond super-resolution enlarger 105 may be equal to or different fromthe resolution after enlargement at the first super-resolution enlarger103.

The second resolution converter 106 is configured for functions toacquire super-resolution enlarged decoded pictures from the secondsuper-resolution enlarger 105, and work on acquired super-resolutionenlarged decoded pictures to implement a process for a prescribedresolution conversion to create, from the super-resolution enlargeddecoded pictures having a resolution higher than the standardresolution, a set of super-resolution decoded pictures (as a set ofsuper-resolution enlarged and converted decoded pictures) such as havingthe standard resolution for instance, to supply to the second encoder107. There may be configuration with a function to have a ratio ofresolution conversion set up upon implementation of the process for theprescribed resolution conversion as preset information on a prescribedresolution conversion ratio, or as acquired information on resolutionconversion ratio such as set up by an external user, and work on thebasis of acquired information on resolution conversion ratio toestablish a resolution conversion ratio to implement the process for theprescribed resolution conversion. More preferably, to provideinformation on the resolution conversion ratio, there should be use of aconfiguration to implement an unshown process for a prescribed entropyencoding, to create a bit sequence of information on resolutionconversion ratio, to supply to the multiplexer 109.

The second encoder 107 is configured for functions to havesuper-resolution enlarged and converted signals of input pictures withthe standard resolution from the first resolution converter 104 asencoding target pictures, decoded pictures from the first encoder 102 asfirst reference pictures, and super-resolution enlarged and convertedsignals of decoded pictures with the standard resolution from the secondresolution converter 106 as second reference pictures, and implementthereon a combination of processes for a prescribed prediction and asecond encoding, to create a second sequence of encoded bits, and supplythe multiplexer 109 with the second sequence of encoded bits thuscreated.

For a reference picture to be used in a process for the prescribedprediction at the second encoder 107, there may be a configuration toemploy either a first reference picture obtained from a decoded pictureor a second reference picture obtained from a super-resolution enlargedand converted decoded picture, to create a predictive picture, andsubtract the predictive picture from a target picture, to create a dataon a difference in between.

There may be a configuration involved to create a difference data from atarget picture, using neither first reference picture nor secondreference picture.

For control to make a selection of reference picture for each picture orfor each set of a prescribed number of pictures, the second encoder 107may be configured to create a set of data on the selection of referencepicture to identify a first reference picture or a second referencepicture whichever is used, and implement thereon a process for aprescribed entropy encoding using an unshown entropy encoder to create asequence of encoded bits of data on the reference picture selection, tosupply to the multiplexer 109. There are sequences of encoded bits ofdata on reference picture selection, multiplexed as a bit sequence 310of data on encoding parameters constituting part of a multiplexed bitsequence 300, as will be described later on. There may be use ofcombination of target pictures each respectively divided with no spacesleft into regions of a prescribed area and first and second referencepictures likewise divided with no spaces left into regions of aprescribed area, to operate for each commensurate region to identify afirst reference picture or a second reference picture whichever isselective to create a predictive picture, and subtract the predictivepicture from a target picture, to create a data on a difference inbetween. There may be operations made for each region of a prescribedarea to have data for identification of which reference picture has beenused, as data on reference picture selection, and implement thereon aprocess for a prescribed entropy encoding using an unshown entropyencoder, to create, and supply to the multiplexer 109, a resultantsequence of encoded bits of data on reference picture selection. Theremay well be regions of a prescribed area shaped as regions of aprescribed rectangular form, or as regions of an arbitrary formconforming to a prescribed domain division.

The second encoder 107 may be configured for combination of a set ofoperations to subtract a first reference picture as a predictive picturefrom an encoding target picture, to obtain a data on a difference inbetween as a first difference data, and implement thereon a process fora prescribed second encoding to create a first sequence of bits encodedby the second encoding, a set of operations to subtract a secondreference picture as a predictive picture from the encoding targetpicture, to obtain a data on a difference in between as a seconddifference data, and implement thereon the process for the prescribedsecond encoding to create a second sequence of bits encoded by thesecond encoding, and a set of operations to create a data on differencesimply from the encoding target picture, using neither first referencepicture nor second reference picture, to use as a third difference data,and implement thereon the process for the prescribed second encoding tocreate a third sequence of bits encoded by the second encoding, toprovide the multiplexer 109 with thus created first to third sequencesof bits encoded by second encoding. There are data on methods forselection of respective types of reference pictures and data on encodingmethods as described, which may be collected as data on referencepicture selection modes and as data on encoding modes for the secondencoding, respectively, and processed by implementing thereon a processfor a prescribed entropy encoding using an unshown entropy encoder, tosupply to the multiplexer 109.

The second encoder 107 may be configured to execute an encoding in theCGS (Coarse Grain Scalability) layer under MPEG-4 SVC for instance, andwork for sequences of moving pictures equal in spatial resolution to thespatial resolution of sequences of moving pictures having a standardresolution at the first encoder 102, to operate complying with aprescribed syntax structure meeting the restraining conditions tofacilitate a conversion from SVC to AVC in terms of the AVC rewriting bythe JVT (Joint Video Team) being a joint group of MPEG and ITU-T, tomake the second encoding for creation of second encoded bit sequences.This configuration permits the second encoder 107 to create secondencoded bit sequences, as sequences of such encoded bits that afford tomake the conversion from the encoding format of SVC that is ahierarchical encoding to the encoding format of AVC that is a singlelayer encoding, without the need of making such a reencoding thatinvolves a combination of complete decoding and encoding.

The multiplexer 109 is configured for functions to acquire from thefirst encoder 102 a first sequence of encoded bits and from the secondencoder 107 a second sequence of encoded bits, and operate complyingwith a prescribed syntax structure to implement a process ofmultiplexing the first sequence of encoded bits, the second sequence ofencoded bits, and sequences of encoded bits of associated data coveringsets of data on encoding parameters of respective types used in encodingprocesses, involving data on motion vectors and data on quantizingparameters, encompassing data on the above-noted reference pictureselection and the like, as well as sets of data on encoding modes orsuch, as they are each respectively processed through an unshown entropyencoder, while inserting identification data for their identification,to create a sequence of bits multiplexed as illustrated in FIG. 3 thatwill be described later on. The multiplexer 109 may well be configuredfor functions to additionally acquire data on modes of reference pictureselection and data on reference picture selection as prepared at thesecond encoder 107 in the form of sequences of encoded bits encodedthrough an unshown entropy encoder, and implement thereon a process fora multiplexing to create a sequence of bits multiplexed as describedabove.

According to the present embodiment, there is a system including theforegoing configurations and adapted to work on input moving pictures,to implement a process for a prescribed super-resolution enlargement anda process for a prescribed resolution conversion, creatingsuper-resolution enlarge and converted signals of moving pictures inputwith a standard resolution, to make use of them allowing for a movingpicture encoding based on an increased amount of information relative toan information amount of input moving pictures. It is noted that betweenthe first super-resolution enlarger 103 and the second super-resolutionenlarger 105, the enlargement ratios applied may not be always equal toeach other. Likewise, between the first resolution converter 104 and thesecond resolution converter 106, the resolution conversion ratiosapplied may not be always equal to each other. However, forconfiguration in FIG. 1, there should be resolution conversion ratiosestablished for the first resolution converter 104 and the secondresolution converter 106 to have a spatial resolution after theirprocesses for resolution conversion, equalized to the standardresolution.

Description is now made of actions of the moving picture encoding systemaccording to the first embodiment shown in FIG. 1, with reference to aflowchart of FIG. 2.

First, the first accumulation buffer 101 stores therein input movingpictures, as necessary in number of pictures for the firstsuper-resolution enlarger 103 to implement a process for a firstsuper-resolution enlargement (step S201).

The first super-resolution enlarger 103 acquires from the firstaccumulation buffer 101 two or more pictures, as necessary for a processfor a prescribed super-resolution enlargement, and implements thereonthe process for the prescribed super-resolution enlargement (step S202),whereby it creates super-resolution enlarged pictures with a resolutionhigher than a standard resolution that input moving pictures have as aspatial resolution thereof, to supply to the first resolution converter104. For the first super-resolution enlarger 103, specificconfigurations as well as contents of the process for super-resolutionenlargement will be described later on.

After that, the first resolution converter 104 acquires super-resolutionenlarged pictures from the first super-resolution enlarger 103, andimplements thereon a process for a prescribed resolution conversion(step S203), whereby it creates super-resolution enlarged and convertedsignals of pictures input with the standard resolution, and supplies thesecond encoder 107 with thus created super-resolution enlarged andconverted signals of pictures input with the standard resolution.

After that, the second encoder 107 acquires super-resolution enlargedand converted signals of input pictures with the standard resolutionfrom the first resolution converter 104, storing them in a prescribedbuffer for temporary accumulation (step S204).

On the other hand, the first encoder 102 acquires from the firstaccumulation buffer 101 a sequence of moving pictures with the standardresolution as necessary for implementation of a process for a prescribedpre-super-resolution encoding, and implements thereon a process for anencoding and a decoding at the standard resolution (step S205), wherebyit creates a first sequence of encoded bits with the standard resolutionas a result of the encoding process, and a set of decoded pictures withthe standard resolution as a result of the decoding. After that, thefirst encoder 102 works to supply the first sequence of encoded bitsthus created to the multiplexer 109, and further to supply the set ofdecoded pictures thus created to the second super-resolution enlarger105 and the second encoder 107.

After that, the second encoder 107 acquires the set of decoded picturesfrom the first encoder 102, storing in a prescribed buffer for temporaryaccumulation (step S206).

After that, the second super-resolution enlarger 105 works like thefirst super-resolution enlarger 103, to acquire the set of decodedpictures from the first encoder 102, and implement thereon a process fora prescribed super-resolution enlargement (step S207), whereby itcreates a set of super-resolution enlarged decoded pictures with aresolution higher than the standard resolution, to supply to the secondresolution converter 106.

After that, the second resolution converter 106 acquires the set ofsuper-resolution enlarged decoded pictures from the secondsuper-resolution enlarger 105, and implements thereon a process for aprescribed resolution conversion (step S208), whereby it createssuper-resolution enlarged and converted signals of decoded pictures withthe standard resolution, to supply to the second encoder 107.

After that, the second encoder 107 acquires super-resolution enlargedand converted signals of decoded pictures with the standard resolutionfrom the second resolution converter 106, storing them in a prescribedbuffer for temporary accumulation (step S209).

The combination of processes associated with steps S202 to S204 and thecombination of processes associated with steps S205 to S209 may beimplemented in parallel, or in series with either ahead in execution.

With the combinations of processes at steps S202 to S209 completed, thesecond encoder 107 has a set of encoding target pictures in the form ofsuper-resolution enlarged and converted signals of input pictures asthey are input pictures having undergone the first combination ofsuper-resolution enlargement and resolution conversion through the firstsuper-resolution enlarger 103 and the first resolution converter 104, aset of first reference pictures in the form of decoded pictures as theyare input pictures having been simply encoded and decoded at the firstencoder 102, and a set of second reference pictures in the form ofsuper-resolution enlarged and converted signals of decoded pictures asthey are those decoded pictures having undergone the second combinationof super-resolution enlargement and resolution conversion through thesecond super-resolution enlarger 105 and the second resolution converter106, and implements thereon a process for a prescribed second prediction(step S210), whereby the second encoder 107 creates a set of differencedata at the standard resolution. After that, the second encoder 107works on the set of difference data thus created to implement a secondencoding process as a process for a prescribed second encoding (stepS211), whereby it creates a second sequence of encoded bits, to supplyto the multiplexer 109.

After that, the multiplexer 109 acquires a first sequence of encodedbits from the first encoder 102 and a second sequence of encoded bitsfrom the second encoder 107. Then, complying with a prescribed syntaxstructure, it implements a process of multiplexing the first sequence ofencoded bits, the second sequence of encoded bits, and sequences ofencoded bits of associated data covering sets of data on encodingparameters used in encoding processes, involving data on motion vectors,data on quantizing parameters, and data on reference picture selection,as they area each respectively processed through an unshown entropyencoder, while inserting identification data for identification of a setof subsequent sequences of encoded bits and the like (step S212), tocreate a sequence of multiplexed bits. The present embodiment involves aseries of actions to be complete through the foregoing steps.

According to the present invention, there is a moving picture encodingsystem operable through execution of such the steps to create sequencesof encoded bits as necessary.

FIG. 3 illustrates an example of multiplexed bit sequence 300 to beoutput from the multiplexer 109.

As illustrated in FIG. 3, the multiplexed bit sequence 300 is comprisedof multiplexed information including a bit sequence 310 of data onencoding parameters, a first sequence 320 of encoded bits from the firstencoder 102, and a second sequence 330 of encoded bits from the secondencoder 107.

The bit sequence 310 of data on encoding parameters has sequences ofencoded bits stored therein to define data on encoding parameters ofrespective types that among others the first encoder 103 and the secondencoder 107 have employed in their encoding processes, involving data onmotion vectors, data on quantizing parameters, data on associatedencoding modes, and data on the above-noted selection of referencepictures and reference picture selection modes, and a bit sequence 311of identification data inserted to identify the first sequence ofencoded bits 320 and the second sequence of encoded bits 330 asmultiplexed in the multiplexed bit sequence 300. Such the configurationof multiplexed bit sequence 300 is a simple example, and there may beanother example of configuration to have those data required over anentirety of encoding and stored in a bit sequence 310 of data onencoding parameters, those data employed at the first encoder 102 asinherent information to the first encoder 102 and stored in a firstsequence of encoded bits 320, and those data employed at the secondencoder 107 as inherent information to the second encoder 107 and storedin a second sequence of encoded bits 330.

There have been operations described as being parallel processesaccording to the present embodiment, while those processed in parallelmay well be consecutively processed in a configuration operableaccording to the present embodiment.

According to the present embodiment, there may well be operations tokeep the first super-resolution enlarger 103 and the first resolutionconverter 104 from working, to supply pictures input with the standardresolution and stored in the first accumulation buffer 101, assuper-resolution enlarged and converted signals of input pictures withthe standard resolution, to the second encoder 107, to implement thereona process for a prescribed second encoding at the second encoder 107 ina configuration operable according to the present embodiment.

Like this, the provision of a first super-resolution enlarger 103 and asecond super-resolution enlarger 105 combined with the provision of afirst resolution converter 104 and a second resolution converter 106affords for services on super-resolution enlarged pictures that aredefined with the same standard resolution as input pictures but haveinformation on spatial and temporal frequency components increased by aprocess for a first super-resolution enlargement, to implement a processfor a first resolution conversion that limits the frequency band withina range of desirable frequency components, to create super-resolutionenlarged and converted signals of input pictures with the standardresolution.

Thus created super-resolution enlarged and converted signals of inputpictures with the standard resolution afford for services on thesuper-resolution enlarged pictures created with greater amounts ofinformation on frequency components than input moving pictures have, totake in therefrom information on frequency components within a band offrequencies up to an uppermost one that can be expressed by spatialfrequencies on the standard resolution, into the super-resolutionenlarged and converted signals of input pictures with the standardresolution, thus permitting information amounts of frequency componentsthat input moving pictures have to be extended, to implement a processfor a moving picture encoding to create a first sequence of encoded bitswith the standard resolution and a second sequence of encoded bits withthe standard resolution, allowing for a moving picture encoding to beimplemented on bases of information amounts greater than informationamounts that input moving pictures have.

Further, it is afforded to provide services on moving pictures inputwith the standard resolution, to implement a combination of processesfor prescribed encoding and decoding to create decoded pictures with thestandard resolution. Still more, it is afforded to provide services onthe decoded pictures with the standard resolution, to implement aprocess for a prescribed second super-resolution enlargement, creatingsuper-resolution enlarged decoded pictures with a resolution higher thanthe standard resolution, having increased information on spatial andtemporal frequency components, and implement a process for a prescribedsecond resolution conversion on thus created super-resolution enlargeddecoded pictures, creating super-resolution enlarged and convertedsignals of decoded pictures with the standard resolution. Yet more, itis afforded to have thus created super-resolution enlarged and convertedsignals of decoded pictures with the standard resolution, as referencepictures, and employ them together with super-resolution enlarged andconverted signals of input pictures with the standard resolution, toimplement a process for a prescribed prediction in between, followed byimplementing a process for a prescribed second encoding, to create asecond sequence of encoded bits. It is thus permitted to utilizesuper-resolution enlarged and converted signals of decoded pictures withthe standard resolution, as reference pictures, in combination withsuper-resolution enlarged and converted signals of input pictures withthe standard resolution, to make a hierarchical encoding in betweenmaking use of correlations of spatial resolution between identicalresolutions, allowing for a moving picture encoding based on greateramounts of information than input moving pictures have.

It also is possible for the second encoder 107 to work upon creation ofa predictive picture, on one hand to have a first reference picture inthe form of a decoded picture as it is an input picture having beensimply encoded and decoded at the first encoder 102, and on the otherhand to have a second reference picture in the form of a set ofsuper-resolution enlarged and converted signals of a picture as decodedas it represents that decoded picture which has been processed forsecond super-resolution enlargement and resolution conversion throughthe second super-resolution enlarger 105 and the second resolutionconverter 106, to use the first reference picture or the secondreference picture, whichever is selective to create the predictivepicture, to implement thereon a process for an encoding, permitting ahierarchical encoding to be made by utilization of a correlation ofspatial resolution between the decoded picture and a set ofsuper-resolution enlarged and converted signals of the input picturewith the standard resolution or by utilization of a correlation ofspatial resolution between the set of super-resolution enlarged andconverted decoded signals and the set of super-resolution enlarged andconverted signals of the input picture with the standard resolution,whichever is requested, thus allowing for different multiplexed bitsequences to be created to supply to, accumulate at, and/or transmit toa decoding system end.

Also, it is possible to work along creation of predictive pictures, tocontrol selection of reference picture for each picture or for each setof a prescribed number of pictures, permitting an adaptive creation ofpredictive picture in accordance with the image quality of decodedpicture, resulting in an enhanced encoding efficiency.

Moreover, it is possible to provide a configuration for services tocreate a data of reference picture selection for identifying a firstreference picture or a second reference picture whichever is used, tosupply to the multiplexer 109, to multiplex into a multiplexed bitsequence 300, allowing for a facilitated identification of a referencepicture used in the encoding.

Moreover, it is possible to provide a target picture divided with nospaces left into regions each having a prescribed area, and combinationof a first reference picture and a second reference picture likewisedivided into regions each having a prescribed area, and identify thefirst reference picture or the second reference picture whichever isselective for a respective region to create a predictive picture,thereby permitting an adaptive creation of predictive picture inaccordance with the image quality of decoded picture, resulting in anenhanced encoding efficiency.

Further, it is possible to work on moving pictures input with thestandard resolution, to implement a process for prescribed encoding anddecoding to create a first sequence of encoded bits, and implement aprocess for second encoding to create a second sequence of encoded bits,and to operate complying with a prescribed syntax structure, toimplement a process of multiplexing the sequences of encoded bits,together with sequences of encoded bits of data on encoding parametersused in the encoding processes, involving data on motion vectors, dataon quantizing parameters, data on reference picture selection, and dataon encoding modes, while inserting data else such as identification datafor identification of a set of subsequent sequences of encoded bits, asnecessary to create a sequence of multiplexed bits. The sequence ofmultiplexed bits thus created is configured as a single sequence ofencoded bits including both of a result of encoding on inherent inputmoving pictures and a result of encoding on a set of errors in aprediction using information on frequency components as enlarged insuper-resolution pictures having undergone a process forsuper-resolution enlargement combined with a process for resolutionconversion, and is adaptive for services such as transmitting to anunshown external accumulator before recording in a prescribed recordingmedium, or delivering through a network using an unshown externalcommunication system, thereby permitting a moving picture encodingsystem according to the present embodiment to operate on moving pictureswith heavy amounts of information, allowing for efficient encoding,accumulation, and transmission.

Further, it is possible to acquire a first sequence of encoded bits fromthe first encoder 102 and a second sequence of encoded bits from thesecond encoder 107, and implement thereon a process for a prescribedmultiplexing in accordance with data on encoding modes, permitting avariety of encoded bit sequences to be created, affording to make aselective decoding at a decoding system end.

Further, it is possible to operate according to the present embodiment,to keep the first super-resolution enlarger 103 and the first resolutionconverter 104 from working, to supply pictures input with the standardresolution and stored in the first accumulation buffer 101, assuper-resolution enlarged and converted signals of input pictures withthe standard resolution, to the second encoder 107, to implement thereona process for a prescribed second encoding at the second encoder 107 ina configuration operable according to the present embodiment, therebyaffording to have input pictures with the standard resolution asencoding target pictures at the second encoder 107, permitting thesecond encoder 107 to make more efficient use of reference pictures thanin typical processes for inter-layer prediction to create predictivepictures, reducing information amounts of difference data to be created,thus allowing for an enhanced encoding efficiency of the second sequenceof encoded bits, with an enabled encoding process rendered the nearer tooriginal moving pictures with the standard resolution.

Description is now made of a process for a prescribed super-resolutionenlargement applied to the present embodiment according to the presentinvention.

For resolution enhancement of pictures using a super-resolutionenlarging process, there have been generalized techniques includingthose employing low-resolution pictures relatively strongly correlatingwith each other, such as those simply deviated in position, to predict asingle high-resolution picture, there having been many studies reportedin recent years. For instance, there is “Super-Resolution ImageReconstruction, by Sung C. P. and Min K. P.: A Technical Overview, IEEESignal Proc. Magazine, Vol. 26, No. 3, pp. 21-36, 2003”, among others.

Further, there is “Reconstruction of a high-resolution image bysimultaneous registration, restoration, and interpolation oflow-resolution images, by B. C. Tom and A. K. Katsaggelos, Proc. IEEEInt. Conf. Image Processing, Vol. 2, pp. 539-542, 1995” disclosing an ML(Maximum-likelihood) method as a proposal. The ML method has anevaluation function in terms of a square error between a set of pixelvalues of a low-resolution picture estimated from a high-resolutionpicture and a set of actual observed pixel values, and an estimatedpicture as such a high-resolution picture that minimizes the evaluationfunction. This method is a method of implementing a super-resolutionprocess based on the principle of maximum likelihood estimation.

Further, there is “Extraction of high-resolution frames from videosequences, by R. R. Schulz and R. L. Stevenson, IEEE Trans. ImageProcessing, Vol. 5, pp. 996-1011, 1996” disclosing an MAP (Maximum APosterior) method as a proposal. The MAP method makes an estimation ofsuch a high-resolution picture that minimizes an evaluation functionhaving probability information of high-resolution picture added to asquare error. This method is a method for super-resolution process thatmakes the estimation of high-resolution picture as an optimizationproblem making use of some prior information on a high-resolutionpicture, to maximize the posterior probability.

Further, there is “High resolution image recovery from image-planearrays, using convex projections, by H. Stark and P. Oskoui, J. Opt.Soc. Am. A, Vol. 6, pp. 1715-1726, 1989” disclosing a POCS (ProjectionOnto Convex Sets) method as a proposal. The POCS method is a method forsuper-resolution process that writes simultaneous equations on sets ofpixel values of high-resolution picture and low-resolution picture, andsolves them sequentially, to thereby obtain a high-resolution picture.

According to the present embodiment, the process for the prescribedsuper-resolution enlargement may for instance be an application of anysuch super-resolution process as described above.

There will be description made of an example of basic configuration of aprocess for super-resolution enlargement employed in the presentembodiment according to the present invention.

Part (a) of FIG. 4 is a block diagram showing a detailed configurationof the first super-resolution enlarger 103 in the first embodimentaccording to the present invention, Part (b) of FIG. 4, a combination ofconceptual diagrams illustrating a process for super-resolutionenlargement, and FIG. 5, a flowchart of exemplary actions in the processfor super-resolution enlargement.

In Part (a) of FIG. 4, according to the first embodiment, the firstsuper-resolution enlarger 103 includes a positioner 1031, aninterpolator 1032, an estimated picture creator 1033, and a repetitiondeterminer 1034. It is noted that the second the super-resolutionenlarger 105 has a similar configuration.

The positioner 1031 is configured to work in a processing forsuper-resolution enlargement, to acquire combination of a base pictureconstituting a base in therein and one or more observation pictures,such as those illustrated in Section (I) in Part (b) of FIG. 4. It isnow assumed to acquire such a combination of base picture, observationpicture 1, and observation picture 2, as illustrated in Section (I) inPart (b) of FIG. 4. As illustrated in Section (II) in Part (b) of FIG.4, the positioner 1031 is configured for functions to make aregistration or positioning to pixel positions of a resolution higherthan resolutions of observation pictures land 2 and base picture thusacquired, for an interval-unequal sampling to create nonhomogeneous highresolution pictures. Further, it is configured for a function to supplycreated nonhomogeneous high resolution pictures to the interpolator 1032and the repetition determiner 1034. For the creation of nonhomogeneoushigh resolution pictures, preferably, as illustrated in Section (II) inPart (b) of FIG. 4, observation pictures should be each positionedrelative to pixels of the base picture, to arrange in place to have ahighest correlation with the base picture. Further, in application of aresolution to a certain positioning, preferably, the resolution used tomake the positioning should be a higher resolution than would beobtained after an associated process for super-resolution enlargement.For Part (c) of FIG. 4, there will be description made later on incomparison with Part (c) of FIG. 6.

The interpolator 1032 is configured for functions to acquirenonhomogeneous high resolution pictures created at the positioner 1031,and use acquired nonhomogeneous high resolution pictures, forimplementing a process for a prescribed nonhomogeneous interpolation tocreate therefrom interpolated pictures with a desirable high resolution,to supply to the estimated picture creator 1033. Further, theinterpolator 1032 acquires from the repetition determiner 1034 necessaryinformation for control to continue the process. The control informationfor continuation of process may well include a nonhomogeneous estimatedpicture created at the estimated picture creator 1033. The interpolator1032 may be configured to work when implementing a repetition process inaccordance with the control information, to operate following aprescribed updating method, for use of a nonhomogeneous estimatedpicture to update a nonhomogeneous high resolution picture acquired fromthe positioner 1031, while correcting respective interpolated pixels.

The estimated picture creator 1033 is configured for functions toacquire interpolated pictures created at the interpolator 1032, andimplement thereon a process for a prescribed reconstruction inconsideration of a point spread (PSF) function obtained from aprescribed camera model, implementing a process for a prescribed noiseremoval, as necessary, to create such estimated pictures with adesirable resolution as illustrated in Section (III) in Part (b) of FIG.4. Further, it is configured for functions to implement a re-samplingprocess on estimated pictures, to create nonhomogeneous estimatedpictures corresponding to pixels of nonhomogeneous high resolutionpictures, to supply to the repetition determiner 1034. Further, it isconfigured for functions to acquire control information from therepetition determiner 1034, and operate in accordance with the controlinformation, to create and externally output estimated pictures as highresolution pictures with a desirable resolution.

The repetition determiner 1034 is configured for functions to acquire anonhomogeneous high resolution picture from the positioner 1031 and anonhomogeneous estimated picture from the estimated picture creator1033, for employment of the acquired pictures to follow a prescribeddetermination method to determine whether or not a repetition of theprocess for super-resolution enlargement is necessary, and workdepending on a result thereof to operate as the repetition is necessary,to provide the interpolator 1032 with information for a control tocontinue the process, and operate as the repetition is unnecessary, toprovide the estimated picture creator 1033 with information for acontrol to output estimated pictures with a high resolution as a resultof the process for super-resolution enlargement.

For a process for super-resolution enlargement required to implement ata still higher rate, there may be application of such a rate increasingmethod for super-resolution process as disclosed in Japanese Patent No.3837575 for instance, to a picture processing system according to thepresent invention, to thereby attain a still better configuration.

Description is now made of actions of the super-resolution enlarger 103shown in Part (a) of FIG. 4, with reference to a flowchart in FIG. 5.

First, the positioner 1031 acquires a base picture as a base in anassociated process for super-resolution enlargement, and one or moreobservation pictures (S500). After that, for each picture acquired, itmakes a positioning at a resolution higher than a desirable resolution(S510). On this occasion, after arrangement of pixels of the basepicture to corresponding positions, it arranges pixels of observationpictures, making highest correlations with the base picture, therebycreating an interval-unequally sampled nonhomogeneous high resolutionpicture.

After that, the interpolator 1032 acquires the created nonhomogeneoushigh resolution picture, and implements thereon a process for aprescribed nonhomogeneous interpolation to effect interpolation ofpixels (S520). This creates an interpolated picture with a desirablehigh resolution.

Next, the estimated picture creator 1033 acquires the interpolatedpicture, and implements thereon a process for a prescribedreconstruction, implementing a process for a prescribed noise removal,as necessary, to create an estimated picture with a desirable resolution(S530). Further, it implements a re-sampling process on the estimatedpicture, to create a nonhomogeneous estimated picture corresponding to aset of pixels of the nonhomogeneous resolution picture.

Next, the repetition determiner 1034 acquires the nonhomogeneousresolution picture from the positioner 1031, while acquiring thenonhomogeneous estimated picture from the estimated picture creator1033. Employing the acquired pictures, it follows a prescribeddetermination method, thereby determining whether or not a repetition ofthe process for super-resolution enlargement is necessary (S540).

If the repetition of the process for super-resolution enlargement isdetermined as being necessary (YES at S540), it provides theinterpolator 1032 with information for a control to continue theprocess. If the repetition of the process for super-resolutionenlargement is determined as being unnecessary (NO at S540), it providesthe estimated picture creator 1033 with information for a control tooutput an estimated picture with a high resolution as a result of theprocess for super-resolution enlargement, whereby the estimated picturecreator 1033 operates in accordance with the control information, tooutput the estimated picture created with the desirable resolution, as ahigh resolution picture obtained by the process for super-resolutionenlargement (S550). Then, according to the first embodiment, the processfor super-resolution enlargement goes to an end.

Description is now made of an example of process for a prescribedresolution conversion applied to the first resolution converter 104 aswell as to the second resolution converter 106.

Part (a) of FIG. 6 is a block diagram showing an example ofconfiguration of the first resolution converter 104.

In Part (a) of FIG. 6, the first resolution converter 104 includes apixel inserter 1041, a filtering processor 1042, and a pixel thinner1043. It is noted that among others the second resolution converter 106is likewise configured, as well as a third resolution converter 110 in alater-described embodiment.

The pixel inserter 1041 is adapted for enlargement of resolution of aninput picture to extend the band of spatial frequencies available tohandle, and is configured to insert new pixels between pixels, to createan upsampled picture, to supply to the filtering processor 1042.

The filtering processor 1042 is configured to acquire the upsampledpicture from the pixel inserter 1041, and implement thereon a processfor a filtering using a prescribed low-pass filter, to create aband-limited picture, to supply to the pixel thinner 1043. The processfor the filtering using a prescribed low-pass filter may well be afiltering process implemented on an upsampled picture, by using alow-pass filter adapted for band limitation and interpolation, within aband of spatial frequencies that can represent the spatial resolution ofa picture finally output at the resolution converter, and the filterused may be, for instance, a filter based on the sine function orLanczos function, or an RR filter or FIR filter designed with aprecipitous cut-off frequency characteristic for a frequency response tobe flat over an entirety of band. There may be a process reduced in dutyby use of such a filter as based on the Nearest-neighbor method, theBilinear method, or the Spline method.

The pixel thinner 1043 is configured to acquire the band-limited picturefrom the filtering processor 1042, and implement thereon a process for aprescribed pixel thinning or decimation for a match with the resolutionof a picture finally output at the resolution converter, to create apicture to be output.

Part (b) of FIG. 6 is a combination of conceptual diagrams illustratinghow signal components vary in a picture subjected to a contractionprocess by decimation after a process for enlargement of picture byresolution conversion at a resolution converter such as the firstresolution converter 104.

As illustrated in Section (I) in Part (b) of FIG. 6, assuming an inputpicture inherently containing a range of spatial frequency componentsnot exceeding an ‘fa’, in the process for resolution conversion at theresolution converter, even if this was operated to implement a processfor enlargement of the picture with a resolution enhanced to enlarge therange of spatial frequency components available to handle, and created apicture with an increased number of pixels, the range of spatialfrequency components contained therein as itself could not haveincreased instead. Therefore, as illustrated in Section (II) in Part (b)of FIG. 6, there would have been resulted a state simply containing therange of spatial frequency components left as it was, so even if thiswas subjected to a low-pass filtering process to limit the band to an‘fmax’, still combined with a process for an inter-pixel thinning downto the pixel number of an original spatial resolution, there would havebeen created a picture containing the range of spatial frequencycomponents left as it was.

On the other hand, Part (c) of FIG. 4 is a combination of conceptualdiagrams illustrating how signal components vary in a picture subjectedto a decimation process after a process for super-resolution enlargementat a super-resolution enlarger such as the first super-resolutionenlarger 103.

As illustrated in Section (I) in Part (c) of FIG. 4, assuming an inputpicture inherently containing a range of spatial frequency componentsnot exceeding an ‘fa’, in the process for super-resolution enlargementat the super-resolution enlarger that makes such a processing asillustrated in Part (b) of FIG. 4, the range of spatial frequencycomponents contained therein as itself does increase. Therefore, asillustrated in Section (II) in Part (c) of FIG. 4, assuming a resultantstate having a range of contained spatial frequency components extendedup to an ‘fb’, if this is subjected to a low-pass filtering process tolimit the band to an ‘fmax’, there appears a picture created with suchcomponents inclusive that exceed the range of spatial frequencycomponents inherently contained up to the ‘fa’, but are still containedin the picture which can express a range of spatial frequencies up tothe ‘fmax’.

Such the process is implemented for a super-resolution enlargement atthe super-resolution enlarger 103, and provides a picture that is equalin resolution to a resolution conversion at the first resolutionconverter 104, but is essentially different therefrom in that the formercan create and make use of such a picture as compensated for thosepicture components to be inherently contained. This is similar also inrelationships such as associated with the second super-resolutionenlarger 105 or the third super-resolution enlarger 110.

There may be configuration with a function to have a ratio of resolutionconversion set up upon implementation of the process for the prescribedresolution conversion as preset information on a prescribed resolutionconversion ratio, or as acquired information on resolution conversionratio such as set up by an external user, and work on the basis ofacquired information on resolution conversion ratio to establish aresolution conversion ratio to implement the process for the prescribedresolution conversion.

For instance, assuming among others the first resolution converter 104or the second resolution converter 106 configured as illustrated in Part(a) of FIG. 6, this operates upon a process being implemented for adecimation of picture from a super-resolution enlarged picture with aresolution higher than a standard resolution to a super-resolutionenlarged and converted picture with the standard resolution, to set upcombination of a proportion of new pixels to be inserted between pixelsat the pixel inserter 1041 and a proportion of pixels to be thinned outat the pixel thinner 1043, depending on a given resolution conversionratio.

The resolution conversion ratio can be expressed such that theresolution conversion ratio=a pixel insertion ratio/a pixel thinningratio, where the pixel insertion ratio is such that pixel insertionratio=input pixels/output pixels, and the pixel thinning ratio is suchthat pixel thinning ratio=input pixels/output pixels.

For instance, assuming a super-resolution enlarged picture created fromthe standard resolution with a resolution conversion ratio of 2, thereare operations to return the picture to the standard resolution with aresolution conversion ratio of 1/2, including a setting made at thepixel inserter 1041, such that input pixels:output pixels=1:1, i.e., tohave an input picture supplied, as it is, in the form of an upsampledpicture to the filtering processor 1042. At the pixel thinner 1043,there is a setting made such that input pixels:output pixels=2:1, i.e.,to implement a process of thinning one pixel out of each set of twopixels to create a picture to be output.

Further, to implement a process for enlargement of picture using thefirst resolution converter 104 configured as illustrated in Part (a) ofFIG. 6, assuming creating a picture enlarged twice the standardresolution with a resolution conversion ratio of 2=2/1, for instance,there are operations including a setting made at the pixel inserter 1041such that input pixels:output pixels=1:2, i.e., to insert one pixel perone pixel to create an upsampled picture to supply to the filteringprocessor 1042. At the pixel thinner 1043, there is a setting made suchthat input pixels:output pixels=1:1, i.e., to have a picture output asit is to output without implementing the thinning process.

It is noted that the configuration of first resolution converter 104illustrated in Part (a) of FIG. 6 is a simple example, and there may bean example configured as necessary for a process for enlargement orcontraction to a more flexible resolution, such as by use of the Splinemethod, to prepare from an input picture a set of pixels arranged inarbitrary positions with a fractal positional precision, to create apicture with a desirable resolution.

Therefore, according to the first embodiment, there is a moving pictureencoding system adapted to set a spatial resolution that an input movingpicture has as a standard resolution, and process information onfrequency components in the spatial direction and the temporal directionthat has been potentially contained in the input moving picture butunable to express to a sufficient degree by the standard resolution, toreconstruct on basis of a prescribed super-resolution process at a firstsuper-resolution enlarger 103, describing with a resolution higher thanthe standard resolution, to create a super-resolution enlarged picturewith an extended amount of information of the information on frequencycomponents the input moving picture has, to implement thereon a movingpicture encoding process, permitting an encoding of input movingpictures based on a greater amount of information than the movingpictures have, as an advantageous effect.

Moreover, according to the first embodiment, there is a moving pictureencoding system adapted to operate on a super-resolution enlargedpicture created at a first super-resolution enlarger 103, to implement aprocess for a prescribed resolution conversion at a first resolutionconverter 104 to create a super-resolution enlarged and convertedpicture described with a standard resolution, thus creating informationon frequency components in the spatial direction and the temporaldirection that has been potentially contained in an input moving picturebut unable to express to a sufficient degree by the standard resolution,while processing information on frequency components that have not beeninherently contained, to restrict within a range of information onfrequency components that can be expressed with the standard resolution,to include in the super-resolution picture, thereby reflecting on thesuper-resolution picture such information on frequency components thathave been unable to express in the input moving picture, to implement amoving picture encoding process on the super-resolution picture,permitting an encoding of input moving pictures based on a greateramount of information than the moving pictures have, as an advantageouseffect.

Further, according to the first embodiment, there is a moving pictureencoding system adapted to operate on a decoded picture with a standardresolution as encoded and decoded to create at a first encoder 102, toimplement a combination of processes for prescribed super-resolutionenlargement and resolution conversion at a second super-resolutionenlarger 105 and a second resolution converter 106, respectively,creating a super-resolution enlarged decoded picture and asuper-resolution enlarged and converted signal of the decoded picturewith the standard resolution, and operate at a second encoder 107, asthis is given an input picture after another combination of processesfor super-resolution enlargement and resolution conversion at a firstsuper-resolution enlarger 103 and a first resolution converter 104,respectively, to have a super-resolution enlarged and converted signalof the input picture, as an encoding target picture, the decoded picturebeing the input picture encoded and decoded as it is at the firstencoder 102, as a first reference picture, and the super-resolutionenlarged and converted signal of the decoded picture that is the decodedpicture after the combination of processes for super-resolutionenlargement and resolution conversion at the second super-resolutionenlarger 105 and the second resolution converter 106, as a secondreference picture, to implement thereon a combination of processes forprescribed second prediction and encoding, thus affording to obtain apredictive picture or predictive blocks with an image quality betterthan predictive pictures or predictive blocks obtainable by motionpicture encoding techniques in the past, with resultant enhancement inencoding efficiency, as an advantageous effect.

(Second Embodiment)

In the first embodiment described above, there has been a second encoder107 working as illustrated in FIG. 1 to have a pair of referencepictures input thereto, that is, combination of a first referencepicture being an input picture as encoded and decoded as it is at afirst encoder 102 to provide as a decoded picture, and a secondreference picture being the decoded picture as subjected to acombination of second super-resolution enlargement and resolutionconversion through a second super-resolution enlarger 105 and a secondresolution converter 106 to provide as a super-resolution enlarged andconverted signal of the decoded picture, to employ the referencepictures enabling a high efficient prediction to create a predictivepicture, whereas there may well be an example adapted for instance, asillustrated in FIG. 7, for operation to supply a second encoder 107simply with a first reference picture being an input picture as encodedand decoded as it is at a first encoder 102 to provide as a decodedpicture, or as illustrated in FIG. 8, for operation to supply a secondencoder 107 simply with a second reference picture being a decodedpicture from a first encoder 102 as subjected to a combination of secondsuper-resolution enlargement and resolution conversion through a secondsuper-resolution enlarger 105 and a second resolution converter 106 toprovide as a super-resolution enlarged and converted signal of thedecoded picture, combined with operations for the second encoder 107 toemploy simply the first or the second reference picture to make acombination of prediction and encoding using an encoding target picturebeing a super-resolution enlarged and converted signal of an inputpicture as subjected to a combination of first super-resolutionenlargement and resolution conversion.

Or else, there may well be an example adapted, as illustrated in FIG. 9,for operations for a second encoder 107 to make a combination ofprediction and encoding using an encoding target picture being asuper-resolution enlarged and converted signal of an input picture assubjected to a combination of first super-resolution enlargement andresolution conversion, excluding operations to supply the second encoder107 with either of two reference pictures, that is, without an operationto supply a first reference picture being an input picture as encodedand decoded as it is at a first encoder 102 to provide as a decodedpicture, and an operation to supply a second reference picture being thedecoded picture as subjected to a combination of second super-resolutionenlargement and resolution conversion through a second super-resolutionenlarger 105 and a second resolution converter 106 to provide as asuper-resolution enlarged and converted signal of the decoded picture.

(Third Embodiment)

Description is now made of a moving picture encoding system according toa third embodiment of the present invention.

FIG. 10 is a block diagram showing an example of configuration of themoving picture encoding system according to the third embodiment.

Referring to FIG. 10, the moving picture encoding system according tothe third embodiment is configured as a specific example of the movingpicture encoding system according to the first embodiment shown in FIG.1, including an accumulation controller 412, a second accumulationbuffer 416, a second reference picture buffer 418, and a code ratecontroller 423 added anew.

In FIG. 10, there is a first encoder 102 including a first predictor402, a first encoder element 403, a first decoder element 404, anintra-predictor 405, a first predictive picture buffer 406, a firstreference picture buffer 407, a motion estimator 408, a motioncompensator 409, an adder 410, and a deblocker 411. This configurationaffords to implement an equivalent function to MPEG-4 AVC specified tostandardize under the ISO/IEC SC29 WG11 as a typical motion pictureencoding. However, this specific configuration of the first encoder 102is an example, so there may well be any example else subject tofunctions of the first encoder 102 to be implemented. For instance,there may be an example compliant with MPEG-2 employed in digitalbroadcasting or the like, or an example compliant with MPEG-4 SVC beinga scalable extension of MPEG-4 AVC. There may be use of apre-super-resolution encoder element based on a wavelet transform, astypified by JPEG2000.

Further, in FIG. 10, there is a second encoder 107 including a thirdaccumulation buffer 415, a second predictor 420, and a second encoderelement 421.

Description is now made of those component elements of the movingpicture encoding system according to the third embodiment shown in FIG.10, which have different functions or new functions relative tocomponent elements of the first embodiment shown in FIG. 1.

The first predictor 402 is configured for functions to acquire from afirst accumulation buffer 101 a set of pictures with a standardresolution as necessary for a process for a prescribed encoding, andfrom the first predictive picture buffer 406 a set of predictivepictures or predictive blocks stored therein, and implement a processfor a prescribed prediction on acquired pictures and predictivepictures, to create data on differences, to supply to the first encoderelement 403. The process for prediction at the first predictor 402 maywell be a process of having a respective one of pictures acquired withthe standard resolution, as an encoding target picture, and subtractingacquired predictive pictures or predictive blocks from the targetpicture, creating data on differences in between, to supply to the firstencoder element 403.

The first encoder element 403 is configured for functions to acquirefrom the first predictor 402 data on differences after the process forprediction, and implement thereon the process for the prescribedencoding, to create a first sequence of encoded bits with the standardresolution, to supply to the first decoder element 404 and to amultiplexer 109. The first encoder element 403 may be configured tosupply a first sequence of encoded bits to the first decoder element404, as illustrated in Part (a) of FIG. 11, or may be configured tocreate data after quantization in a course of the process for theprescribed encoding, to supply to the first decoder element 404, asillustrated in Part (b) of FIG. 11. The first encoder element 403 isconfigured for a function to vary the code rate of a first sequence ofencoded bits being created in the process for encoding, in accordancewith control information from the code rate controller 423.

The first decoder element 404 is configured for functions to work inaccordance with the type of output at the first encoder element 403illustrated in Part (a) or (b) of FIG. 11, to acquire a first sequenceof encoded bits from the first encoder element 403 as illustrated inPart (a) FIG. 12 or acquire post-quantization data from the firstencoder element 403 as illustrated in Part (b) FIG. 12, and implement aprocess for a prescribed decoding on the first sequence of encoded bitsor the post-quantization data, to create decoded data on differences, tosupply to the adder 410.

The intra-predictor 405 is configured for functions to acquire data ondecoded pictures as necessary for a process for a prescribedintra-prediction, and implement thereon the process for the prescribedintra-prediction, to create intra-predictive data, to supply to thefirst predictive picture buffer 406. For creation of intra-predictivedata, the intra-predictor 405 may well implement the process for theprescribed intra-prediction on blocks as decoded about a blockconstituting a target of prediction to create a set of intra-predictiveblocks.

The first predictive picture buffer 406 is configured for functions toacquire intra-predictive data from the intra-predictor 405 and motioncompensation predictive data from the motion compensator 409, toaccumulate as predictive pictures or predictive blocks, and supplypredictive pictures or predictive blocks to the first predictor 402 andthe adder 410. The first predictive picture buffer 406 may be configuredto acquire information on control for accumulation from the accumulationcontroller 412, and work on the accumulation control information tocontrol acquisition, accumulation, and supply of predictive pictures orpredictive blocks.

The first reference picture buffer 407 is configured for functions toacquire deblocked decoded pictures or decoded blocks from the deblocker411 and decoded pictures or decoded blocks from the adder 410, toaccumulate as reference pictures or reference blocks, and supplyaccumulated reference pictures or reference blocks, as necessary, todemanding ends inclusive of the intra-predictor 405, the motionestimator 408, the motion compensator 409, and the second accumulationbuffer 416. The first reference picture buffer 407 may be configured toacquire information on control for accumulation from the accumulationcontroller 412, and work on the accumulation control information tocontrol acquisition, accumulation, and supply of reference pictures orreference blocks.

The motion estimator 408 is configured for functions to acquire encodingtarget pictures from the first accumulation buffer 101 and referencepictures from the first reference picture buffer 407, and makes thereona prescribed estimation of motions to create data on motion vectors. Themotion estimator 408 is configured for a function to supply createdmotion vector data at least to the motion compensator 409. Preferably,there should be a configuration for motion vector data created at themotion estimator 408 to be processed for a prescribed entropy encodingat an unshown entropy encoder, to supply to the multiplexer 109.

The motion compensator 409 is configured for functions to acquirereference pictures from the first reference picture buffer 407 andmotion vector data from the motion estimator 408, and make a prescribedmotion compensation on acquired reference pictures and motion vectordata, to create predictive pictures, to supply to the first predictivepicture buffer 406.

The adder 410 is configured for functions to acquire decoded differencedata from the first decoder element 404 and predictive pictures orpredictive blocks from the first predictive picture buffer 406, and makean addition thereof, to create decoded pictures, to supply to the firstreference picture buffer 407 and the deblocker 411.

The deblocker 411 is configured for functions to acquire decodedpictures from the adder 410, and implement thereon a process for aprescribed deblocking, to create deblocked decoded pictures, to supplyto the first reference picture buffer 407.

Part (a) and Part (b) of FIG. 11 show examples of configuration of thefirst encoder element 403 according to the third embodiment.

In an example, the first encoder element 403 includes an orthogonaltransformer 4031, a quantizer 4032, and an entropy encoder 4033.

The orthogonal transformer 4031 is configured for functions to acquiredata on differences, and implement thereon a process for a prescribedorthogonal transform, to create data on orthogonal transformcoefficients, to supply to the quantizer.

The quantizer 4032 is configured for functions to acquire data onorthogonal transform coefficients from the orthogonal transformer 4031,and implement thereon a process for a prescribed quantization, to createquantized data, to supply to the entropy encoder 4033. For applicationto a model of first decoder element 404 configured to work withoutmaking an entropy decoding on an encoded bit sequence, to acquirequantized data as a result of local decoding and implement thereon aprocess for a prescribed decoding, there may be use of a quantizer 4032configured to supply quantized data to the first decoder element 404, asillustrated in Part (b) of FIG. 11.

The entropy encoder 4033 is configured for functions to acquirequantized data from the quantizer 4032, and implement thereon a processfor a prescribed entropy encoding, to create a first sequence of encodedbits, to supply to the multiplexer 109. For application to a model offirst decoder element 404 configured to make an entropy decoding on afirst encoded bit sequence, to acquire quantized data and implementthereon a process for a prescribed decoding, there may be use of anentropy encoder 4033 configured to create a first sequence of encodedbits, to supply to the first decoder element 404, as illustrated in Part(a) of FIG. 11.

Part (a) and Part (b) of FIG. 12 show examples of configuration of thefirst decoder element 404 according to the third embodiment.

For adaptation to a local decoding process, the first decoder element404 includes at least an inverse quantizer 4042, and an inverseorthogonal transformer 4043, as illustrated in Part (a) and Part (b) ofFIG. 12. For configuration to acquire a first sequence of encoded bitsto implement thereon a complete decoding process, it may additionallyinclude an entropy decoder 4041, as illustrated in Part (a) of FIG. 12.

The entropy decoder 4041 is configured for functions to acquire a firstsequence of encoded bits from the first encoder element 403, andimplement thereon a process for a prescribed entropy decoding, to createquantized data, to supply to the inverse quantizer 4042.

The inverse quantizer 4042 is configured for functions to acquirequantized data from the entropy decoder 4041 or the quantizer 4032, andimplement thereon a process for a prescribed inverse quantization, tocreate inverse quantized data on orthogonal transform coefficients, tosupply to the inverse orthogonal transformer 4043.

The inverse orthogonal transformer 4043 is configured for functions toacquire inverse quantized data on orthogonal transform coefficients fromthe inverse quantizer 4042, and implement thereon a process for aprescribed inverse orthogonal transform, to create decoded data ondifferences, to supply to the adder 410.

Referring again to FIG. 10, according to the third embodiment, thesecond encoder 107 includes the third accumulation buffer 415, thesecond predictor 420, and the second encoder element 421.

The third accumulation buffer 415 is configured for functions to acquirefrom a first resolution converter 104 super-resolution enlarged andconverted signals of input pictures with a standard resolution, toaccumulate therein, and supply to the second predictor 420. The thirdaccumulation buffer 415 is configured for functions to acquire from theaccumulation controller 412 information on control for accumulation, andwork on the accumulation control information to control acquisition,accumulation, and supply of super-resolution enlarged and convertedsignals of input pictures with the standard resolution.

The second predictor 420 is configured for functions to acquire from thethird accumulation buffer 415 super-resolution enlarged and convertedsignals of input pictures with the standard resolution, from the secondaccumulation buffer 416 decoded pictures with the standard resolution,and from a second resolution converter 106 super-resolution enlarged andconverted signals of decoded pictures with the standard resolution, andhave the super-resolution enlarged and converted signals of inputpictures, as encoding target pictures, the decoded pictures derived fromthe first encoder 102, as first reference pictures, and super-resolutionenlarged and converted decoded pictures of the super-resolution enlargedand converted signals of decoded pictures with the standard resolution,as second reference pictures, to implement thereon a process for aprescribed prediction to create predictive pictures, and subtract thepredictive pictures from the target pictures, to create data ondifferences, to supply to the second encoder element 421.

The second encoder element 421 is configured for functions to acquiredata on differences from the second predictor 420, and implement thereona process for a prescribed encoding to create the above-noted secondsequence of encoded bits, to supply to the multiplexer 109, whileworking on control information from the code rate controller 423, tovary the code rate of the second sequence of encoded bits being createdby the encoding process.

The accumulation controller 412 is configured for functions to supplyinformation on control for accumulation to each accumulation buffer, tocontrol among others the state of accumulation, acquisition of pictures,and timings of supply of the accumulation buffer.

The second accumulation buffer 416 is configured for functions toacquire decoded pictures with the standard resolution from the firstreference picture buffer 407, and supply acquired decoded pictures tothe second predictor 420 as well as to a second super-resolutionenlarger 105.

The second accumulation buffer 416 is configured for functions toacquire information on control for accumulation from the accumulationcontroller 412, and work on the accumulation control information tocontrol acquisition, accumulation, and supply of decoded pictures.

The second reference picture buffer 407 is configured for functions toacquire super-resolution enlarged decoded pictures from the secondsuper-resolution enlarger 105, to accumulate therein, and supplyacquired super-resolution enlarged decoded pictures to the secondresolution converter 106. In doing so, the second reference picturebuffer 407 acquires information on control for accumulation from theaccumulation controller 412, and works on the accumulation controlinformation to control acquisition, accumulation, and supply ofsuper-resolution enlarged and converted signals of input pictures withthe standard resolution.

The code rate controller 423 is configured for functions to monitor coderates of encoded bit sequences created at respective encoder elements,to control actions of the encoder elements to have the code rates fallwithin prescribed ranges, and work in accordance with among othersstates of the code rates and states of accumulation buffers obtainedfrom the accumulation controller 412, to control the accumulationcontroller 412 to change actions of the accumulation buffers. The coderate controller 423 is configured to work when controlling actions ofthe encoder elements to control the code rates, to change data onparameters of quantization at the encoder elements to thereby controlthe code rates. There may be an example preferably configured to work onamong others information on states of signal amplitudes of differencedata at each encoder elements and information on reference pictureselection adapted to identify which reference picture is used at eachpredictor, as necessary to determine whether a process forsuper-resolution enlargement on a decoded picture is effective orineffective, to operate when it is ineffective, to command theaccumulation controller 412 to control the second accumulation buffer416 to stop supplying decoded pictures to the second super-resolutionenlarger 105, and make a changeover to use data on parameters ofquantization created as information on predictive differences based onfirst reference pictures, as they are different from those data onparameters of quantization which have been used while the process forsuper-resolution enlargement has been effective, to control the coderates.

Therefore, the moving picture encoding system according to the thirdembodiment is adapted to exhibit similar advantageous effects to themoving picture encoding system according to the first embodiment shownin FIG. 1, and besides configured with components such as anaccumulation controller 412 and a code rate controller 423, implementingamong others accumulation control at respective buffers and code ratecontrol at a first encoder 102 and a second encoder 107, thus permittingan encoding to be executed taking into consideration among othersaccumulation amounts at the buffers and code rates at the first encoder102 and the second encoder 107.

In the third embodiment described, there have been components such as anaccumulation controller 412 and a code rate controller 423 incorporatedin the moving picture encoding system according to the first embodimentshown in FIG. 1, to effect accumulation control at respective buffersand code rate control at a first encoder 102 and a second encoder 107,whereas there may well be an example having components such as anaccumulation controller 412 and a code rate controller 423 incorporatedin the moving picture encoding system according to the second embodimentshown in FIG. 7 as well as in FIG. 8 and FIG. 9, to likewise effectamong others accumulation control and code rate control.

(Fourth Embodiment)

Description is now made of a moving picture encoding system according toa fourth embodiment of the present invention.

FIG. 13 is a block diagram showing an example of configuration of themoving picture encoding system according to the fourth embodiment.

Referring to FIG. 13, the moving picture encoding system according tothe fourth embodiment includes a first accumulation buffer 101, a firstencoder 102, a first super-resolution enlarger 103, a secondsuper-resolution enlarger 105, a third resolution converter 110, a thirdencoder 108, and a multiplexer 109.

The first accumulation buffer 101 is configured for equivalent functionsto the embodiment shown in FIG. 1, and redundant description is omitted.

The first encoder 102 is configured for equivalent functions to theembodiment shown in FIG. 1. In the fourth embodiment, there is a set ofdecoded pictures created to supply to the second super-resolutionenlarger 105 and the third resolution converter 110.

The first super-resolution enlarger 103 is configured for equivalentfunctions to the embodiment shown in FIG. 1. According to the fourthembodiment, the first super-resolution enlarger 103 is configured forfunctions to create super-resolution enlarged pictures with a resolutionhigher than a standard resolution, to supply to the third encoder 108.

The second super-resolution enlarger 105 is configured for equivalentfunctions to the embodiment shown in FIG. 1. According to the fourthembodiment, the second super-resolution enlarger 105 is configured forfunctions to create super-resolution enlarged decoded pictures with aresolution higher than the standard resolution, to supply to the thirdencoder 108.

The third resolution converter 110 is configured for functions toacquire decoded pictures with the standard resolution from the firstencoder 102, and implement a process for a prescribed resolutionconversion on acquired decoded pictures with the standard resolution tocreate resolution conversion enlarged decoded pictures being highresolution enlarged decoded pictures with a resolution equal to thespatial resolution of super-resolution enlarged pictures created at thefirst super-resolution enlarger 103 and that of super-resolutionenlarged decoded pictures created at the second super-resolutionenlarger 105, to supply to the third encoder 108. There may beconfiguration with a function to have a ratio of resolution conversionset up upon implementation of the process for the prescribed resolutionconversion as preset information on a prescribed resolution conversionratio, or as acquired information on resolution conversion ratio such asset up by an external user, and work on the basis of acquiredinformation on resolution conversion ratio to establish a resolutionconversion ratio to implement the process for the prescribed resolutionconversion.

The third encoder 108 is configured for functions to acquire from thefirst super-resolution enlarger 103 super-resolution enlarged pictures,from the third resolution converter 110 resolution conversion enlargeddecoded pictures with a high resolution, and from the secondsuper-resolution enlarger 105 super-resolution enlarged decoded pictureswith the high resolution, to have acquired super-resolution enlargedpictures as encoding target pictures, resolution conversion enlargeddecoded pictures from the third resolution converter 110 as firstreference pictures, and super-resolution enlarged decoded pictures fromthe second super-resolution enlarger 105 as second reference pictures,and implement thereon a combination of processes for prescribedprediction and encoding to create a third sequence of encoded bits, andsupply the multiplexer 109 with the third sequence of encoded bits thuscreated.

For a reference picture to be used in a process for the prescribedprediction at the third encoder 108, there may be a configuration toemploy either a first reference picture obtained from a resolutionconversion enlarged decoded picture or a second reference pictureobtained from a super-resolution enlarged decoded picture, to create apredictive picture, and subtract the predictive picture from a targetpicture, to create a data on a difference in between.

There may be a configuration involved to create a difference data from atarget picture, using neither first reference picture nor secondreference picture.

For control to make a selection of reference picture for each picture orfor each set of a prescribed number of pictures, the third encoder 108may be configured to create a set of data on the selection of referencepicture to identify a first reference picture or a second referencepicture whichever is used, and implement thereon a process for aprescribed entropy encoding using an unshown entropy encoder to create asequence of encoded bits of data on the reference picture selection, tosupply to the multiplexer 109.

There may be use of combination of target pictures each respectivelydivided with no spaces left into regions of a prescribed area and firstand second reference pictures likewise divided with no spaces left intoregions of a prescribed area, to operate for each commensurate region toidentify a first reference picture or a second reference picturewhichever is selective to create a predictive picture, and subtract thepredictive picture from a target picture, to create a data on adifference in between. There may be operations made for each region of aprescribed area to have data for identification of which referencepicture has been used, as data on reference picture selection, andimplement thereon a process for a prescribed entropy encoding using anunshown entropy encoder, to create, and supply to the multiplexer 109, aresultant sequence of encoded bits of data on reference pictureselection. There may well be regions of a prescribed area shaped asregions of a prescribed rectangular form, or as regions of an arbitraryform conforming to a prescribed domain division.

The third encoder 108 may be configured for combination of a set ofoperations to subtract a first reference picture as a predictive picturefrom an encoding target picture, to obtain a data on a difference inbetween as a first difference data, and implement thereon a process fora prescribed third encoding to create a first sequence of bits encodedby the third encoding, a set of operations to subtract a secondreference picture as a predictive picture from the encoding targetpicture, to obtain a data on a difference in between as a seconddifference data, and implement thereon the process for the prescribedthird encoding to create a second sequence of bits encoded by the thirdencoding, and a set of operations to create a data on difference simplyfrom the encoding target picture, using neither first reference picturenor second reference picture, to use as a third difference data, andimplement thereon the process for the prescribed third encoding tocreate a third sequence of bits encoded by the third encoding, toprovide the multiplexer 109 with thus created first to third sequencesof bits encoded by third encoding. There are data on methods forselection of respective types of reference pictures and data on encodingmethods as described, which may be collected as data on referencepicture selection modes and as data on encoding modes for the thirdencoding, respectively, and processed by implementing thereon a processfor a prescribed entropy encoding using an unshown entropy encoder, tosupply to the multiplexer 109.

The multiplexer 109 is configured for functions to acquire from thefirst encoder 102 a first sequence of encoded bits and from the thirdencoder 108 a third sequence of encoded bits, and operate complying witha prescribed syntax structure to implement a process of multiplexing thefirst sequence of encoded bits, the third sequence of encoded bits, andsequences of encoded bits of sets of data used in encoding processes,involving data on motion vectors, data on quantizing parameters, data onreference picture selection, and data on encoding parameters, as theyare each respectively processed through an unshown entropy encoder,while inserting identification data for identification of a set ofsubsequent sequences of encoded bits and the like, to create a sequenceof multiplexed bits to be output. The multiplexer 109 may well beconfigured for functions to additionally acquire data on modes ofreference picture selection and data on reference picture selection ascreated at the third encoder 108 in the form of sequences of encodedbits encoded through an unshown entropy encoder, and implement thereon aprocess for a multiplexing to create a sequence of bits multiplexed asdescribed above.

The multiplexer 109 may be configured to acquire from the first encoder102 a first sequence of encoded bits, and from the third encoder 108 thefirst to the third sequence of bits encoded by the third encoding. Themultiplexer 109 may then be configured to reproduce the acquired firstsequence of encoded bits with the standard resolution, to create a firstto a third sequence of bits encoded by a first encoding with thestandard resolution. The multiplexer 109 may then be configured tooperate complying with a prescribed syntax structure to implement aprocess of multiplexing a respective one of combination of the firstsequence of bits encoded by the third encoding and the first sequence ofbits encoded by the first encoding with the standard resolution,combination of the second sequence of bits encoded by the third encodingand a second sequence of bits encoded by the first encoding with thestandard resolution, and combination of the third sequence of bitsencoded by the third encoding and the third sequence of bits encoded bythe first encoding with the standard resolution, and sequences ofencoded bits of sets of data used in encoding processes, involving dataon motion vectors, data on quantizing parameters, data on referencepicture selection, and data on encoding parameters, as they are eachrespectively processed through an unshown entropy encoder, whileinserting identification data for identification of a set of subsequentsequences of encoded bits and the like, to create a first to a thirdsequence of encoded bits as multiplexed to output. The multiplexer 109may then be configured to operate in response to a demand such as from adata on encoding mode, to selectively create any of the first to thethird sequence of encoded bits as multiplexed. The data on encoding modethen used may be formatted to include at least information to controlthe multiplexing.

According to the fourth embodiment, there is a moving picture encodingsystem configured for the foregoing functions, and adapted to work oninput moving pictures, to implement a process for a prescribedsuper-resolution enlargement and a process for a prescribed resolutionconversion, creating super-resolution enlarged and converted pictures,to make use of them allowing for a moving picture encoding based on anincreased amount of information relative to an information amount ofinput moving pictures. In the configuration shown in FIG. 13, the firstsuper-resolution enlarger 103 and the second super-resolution enlarger105 have enlargement ratios applied thereto, which may well beenlargement ratios equal to each other, while they may be unequal toeach other. Further, the resolution conversion ratio applied to thethird resolution converter 110 may well be such a resolution conversionratio that would provide the same resolution as the spatial resolutionof super-resolution enlarged pictures created at the firstsuper-resolution enlarger 103 and super-resolution enlarged decodedpictures created at the second super-resolution enlarger 105, while itmay not be so.

Description is now made of actions of the moving picture encoding systemaccording to the fourth embodiment shown in FIG. 13, with reference to aflowchart of FIG. 14.

FIG. 14 is a flowchart showing actions of the moving picture encodingsystem according to the fourth embodiment.

First, in the moving picture encoding system according to the fourthembodiment, the first accumulation buffer 101 stores therein inputmoving pictures, as necessary in number of pictures for the firstsuper-resolution enlarger 103 to implement a process for a firstsuper-resolution enlargement (S1401).

The first super-resolution enlarger 103 acquires from the firstaccumulation buffer 101 two or more pictures, as necessary for a processfor a prescribed first super-resolution enlargement, and implementsthereon the process for the prescribed super-resolution enlargement(S1402), whereby it creates super-resolution enlarged pictures with aresolution higher than a standard resolution that input moving pictureshave as a spatial resolution thereof, to supply to the third encoder108.

The third encoder 108 acquires super-resolution enlarged pictures fromthe first super-resolution enlarger 103, storing them in a prescribedbuffer for temporary accumulation (S1403).

On the other hand, the first encoder 102 acquires from the firstaccumulation buffer 101 a sequence of moving pictures with the standardresolution as necessary for implementation of a process for a prescribedpre-super-resolution encoding, and implements thereon a process for anencoding and a decoding at the standard resolution (S1404), whereby itcreates a first sequence of encoded bits with the standard resolution asa result of the encoding process, and a set of decoded pictures with thestandard resolution as a result of the decoding.

After that, the first encoder 102 works to supply the first sequence ofencoded bits thus created to the multiplexer 109, and store the set ofdecoded pictures thus created in a prescribed buffer for temporaryaccumulation (S1405).

The second super-resolution enlarger 105 acquires the set of decodedpictures from the first encoder 102, and implements thereon a processfor a prescribed super-resolution enlargement (S1406), whereby itcreates a set of super-resolution enlarged decoded pictures with aresolution higher than the standard resolution, to supply to the thirdencoder 108.

After that, the third encoder 108 acquires super-resolution enlargeddecoded pictures from the second super-resolution enlarger 105, storingthem in a prescribed buffer for temporary accumulation (S1407).

Further, the third resolution converter 110 acquires the set of decodedpictures from the first encoder 102, and implements thereon a processfor a prescribed resolution conversion (S1408), whereby it createsresolution conversion enlarged decoded pictures being high resolutionenlarged decoded pictures with a resolution higher than the standardresolution, to supply to the third encoder 108.

After that, the third encoder 108 acquires resolution conversionenlarged decoded pictures with a high resolution from the thirdresolution converter 110, storing them in a prescribed buffer fortemporary accumulation (S1409).

The combination of processes associated with steps S1402 to S1403, thecombination of processes associated with steps S1404 to S1407, and thecombination of processes associated with steps S1408 to S1409 may beimplemented in parallel, or in series.

With the foregoing processes completed, the third encoder 108 hassuper-resolution enlarged pictures acquired from the firstsuper-resolution enlarger 103 as encoding target pictures, resolutionconversion enlarged decoded pictures acquired from the third resolutionconverter 110 as first reference pictures, and super-resolution enlargeddecoded pictures acquired from the second super-resolution enlarger 105as second reference pictures, and implements thereon a process for aprescribed third prediction (S1410), whereby it creates data ondifferences at the high resolution. After that, it works on the createddata on differences, to implement a third encoding process that is aprescribed third encoding (S1411), whereby it creates a third sequenceof encoded bits, to supply to the multiplexer 109.

After that, the multiplexer 109 acquires a first sequence of encodedbits from the first encoder 102 and a third sequence of encoded bitsfrom the third encoder 108, and works complying with a prescribed syntaxstructure, to implement a process of multiplexing them together withsets of data used in encoding processes, involving data on motionvectors, data on quantizing parameters, data on reference pictureselection, and data on encoding parameters, as they area eachrespectively processed through an unshown entropy encoder, whileinserting identification data for identification of a set of subsequentsequences of encoded bits and the like (S1412), to create such asequence of multiplexed bits as illustrated in FIG. 15. The presentembodiment involves a series of actions to be complete through theforegoing steps.

FIG. 15 is a diagram of data format illustrating an example of structureof a multiplexed bit sequence 300 created at the multiplexer 109according to the fourth embodiment.

The multiplexed bit sequence 300 has multiplexed therein a bit sequence310 of data on encoding parameters, a first sequence 320 of encoded bitsfrom the first encoder 102, and a third sequence 340 of encoded bitsfrom the third encoder 108.

There have been operations described as being parallel processesaccording to the fourth embodiment, while those processed in parallelmay well be consecutively processed in a configuration operableaccording to the present embodiment.

According to the fourth embodiment, like the first embodiment described,the moving picture encoding system is configured with the firstsuper-resolution enlarger 103 and the second super-resolution enlarger105 working to implement processes for a first and a secondsuper-resolution enlargement, affording to have a spatial resolution ofinput pictures as a standard resolution, permitting the process for thefirst super-resolution enlargement to create super-resolution enlargedpictures with a resolution higher than the standard resolution.

As a result, the process for the first super-resolution enlargement isallowed to process information on spatial and temporal frequencycomponents that have been potentially contained in input moving picturesbut unable to express to a sufficient degree by the standard resolution,to reconstruct from the input moving pictures on the basis of aprescribed super-resolution process, to describe with a resolutionhigher than the standard resolution. It is thus possible to extendinformation on frequency components that input moving pictures have, toimplement thereon a process for moving picture encoding, to make amoving picture encoding based on a greater amount of information thaninput moving pictures have.

Further, it is possible to work on moving pictures input with thestandard resolution, to implement a process for prescribed encoding anddecoding to create decoded pictures with the standard resolution, andimplement a process for a prescribed second super-resolution enlargementon the decoded pictures to create super-resolution enlarged decodedpictures with a resolution higher than the standard resolution withenlarged information on spatial and temporal frequency components, andto have thus created super-resolution enlarged decoded pictures asreference pictures with a high resolution, and use to implement aprocess for a prescribed prediction relative to super-resolutionenlarged pictures, before implementing a process for a prescribed thirdencoding to create a third sequence of encoded bits. Like this, therecan be use of super-resolution enlarged decoded pictures as referencepictures, to effect a hierarchical encoding making use of correlationsof spatial resolution relative to super-resolution enlarged pictures, tomake a moving picture encoding based on a greater amount of informationthan input moving pictures have.

That is, according to the fourth embodiment, like the first embodimentdescribed, such information on frequency components of pictures thatcould have been expressed simply within a range that can be describedwith an inherent spatial resolution of standard resolution as input, isreconstructed making use of correlations that the pictures themselvespotentially contain, so it is possible to implement a process for aprescribed super-resolution enlargement to create anew such informationon frequency components that can be described simply within a range ofspatial resolutions higher than the standard resolution, and reflect insuper-resolution enlarged pictures, permitting differences between thesuper-resolution enlarged pictures being new final encoding targets thuscreated and information on frequency components of spatial resolutionrespectively contained in sequences of original moving pictures with thestandard resolution, to be encoded and transmitted according to thepresent invention, as an essential point. Instead, even with a movingpicture decoding system adapted to simply implement a process for asecond super-resolution enlargement to create super-resolution enlargeddecoded pictures to output to an external display system, there wouldhave been a prescribed decoding process implemented on a first sequenceof encoded bits as decoding targets with a standard resolution to createdecoded pictures with the standard resolution, which should havecontained not a little deterioration in the encoding, so even if theprocess for the second super-resolution enlargement were implemented asprescribed on the decoded pictures, the process implemented should havebeen for a super-resolution enlargement on decoded pictures with thestandard resolution containing encoding deteriorations, and could notalways have been a proper process implemented for super-resolutionenlargement, while it might have had a certain effect. According to thepresent invention, there is a configuration adapted to implement aprocess for a prescribed extension encoding on super-resolution enlargeddecoded pictures being not always sufficient as described, to encode andtransmit their differences relative to information on frequencycomponents contained in super-resolution enlarged pictures as inherentlywould be, thus allowing for more and accurate information on frequencycomponents contained in super-resolution enlarged pictures to besupplied to a decoding system end.

Further, according to the fourth embodiment, it is possible to work uponcreation of a predictive picture, to use a first reference picture fromthe third resolution converter 110 or a second reference picture fromthe second super-resolution enlarger 105, whichever is selective toimplement a third encoding process that is a prescribed third encoding,permitting a hierarchical encoding to be made by utilization of acorrelation of spatial resolution between a super-resolution enlargedpicture and a resolution conversion enlarged decoded picture being thefirst reference picture or by utilization of a correlation of spatialresolution between the super-resolution enlarged picture and asuper-resolution enlarged decoded picture being the second referencepicture, whichever is requested, thus allowing for different multiplexedbit sequences to be created to supply to, accumulate at, and/or transmitto a decoding system end.

Further, according to the fourth embodiment, it is possible to workalong creation of predictive pictures, to control selection of referencepicture for each picture or for each set of a prescribed number ofpictures, permitting an adaptive creation of predictive picture inaccordance with the image quality of decoded picture, resulting in anenhanced encoding efficiency.

Further, according to the fourth embodiment, it is possible to provide aconfiguration for services to create a data of reference pictureselection for identifying a first reference picture or a secondreference picture whichever is used, to supply to the multiplexer 109,allowing for a facilitated identification of a reference picture used inan encoding according to the present invention.

Further, according to the fourth embodiment, it is possible to provide atarget picture divided with no spaces left into regions each having aprescribed area, and combination of a first reference picture and asecond reference picture likewise divided into regions each having aprescribed area, and identify the first reference picture or the secondreference picture whichever is selective for a respective region tocreate a predictive picture, thereby permitting an adaptive creation ofpredictive picture in accordance with the image quality of decodedpicture, resulting in an enhanced encoding efficiency.

Further, it is possible to work on moving pictures input with thestandard resolution, to implement a process for prescribed encoding anddecoding to create a first sequence of encoded bits, and implement aprocess for third encoding to create a third sequence of encoded bitswith the standard resolution, and to operate complying with a prescribedsyntax structure, to implement a process of multiplexing the sequencesof encoded bits, together with sequences of encoded bits of sets of dataused in encoding processes, involving data on motion vectors, data onquantizing parameters, data on reference picture selection, and data onencoding parameters, as they are each respectively processed through anunshown entropy encoder, while inserting identification data foridentification of a set of subsequent sequences of encoded bits and thelike, to create a sequence of multiplexed bits. The sequence ofmultiplexed bits thus created is configured as a single sequence ofencoded bits including both of a result of encoding on inherent inputmoving pictures and a result of encoding on a set of errors in aprediction using information on frequency components as enlarged insuper-resolution enlarged pictures having undergone a process forsuper-resolution enlargement, and is adaptive for services such astransmitting to an unshown external accumulator before recording in aprescribed recording medium, or delivering through a network using anunshown external communication system, thereby permitting a movingpicture encoding system according to the present embodiment to operateon moving pictures with heavy amounts of information, allowing forefficient encoding, accumulation, and transmission.

Further, it is possible to work to acquire a first and a second sequenceof bits encoded by the third encoding, identify a set of sequences ofencoded bits to be multiplexed in accordance with data on encodingmodes, and implement a process for a prescribed multiplexing onidentified sequences of encoded bits to create a sequence of multiplexedbits, permitting a variety of encoded bit sequences to be created,affording to make a selective decoding at a decoding system end.

(Fifth Embodiment)

In the fourth embodiment described above, there has been a third encoder108 working as illustrated in FIG. 13 to have a pair of referencepictures input thereto, that is, combination of a first referencepicture being a decoded picture from a first encoder 102 as processedfor a third resolution conversion at a third resolution converter 110 toprovide as a resolution conversion enlarged decoded picture, and asecond reference picture being the decoded picture from the firstencoder 102 as processed for a second super-resolution enlargement at asecond super-resolution enlarger 105 to provide as a super-resolutionenlarged decoded picture, to employ the reference pictures enabling ahigh efficient prediction to create a predictive picture, whereas theremay well be an example adapted for instance as illustrated in FIG. 16,for operation to simply input a resolution conversion enlarged decodedpicture from a third resolution converter 110, as a reference picture,or for instance as illustrated in FIG. 17, for operation to simply inputa super-resolution enlarged decoded picture from a secondsuper-resolution enlarger 105, as a reference picture, and have asuper-resolution enlarged picture from a first super-resolution enlarger103, as an encoding target picture, for use to make a combination ofprediction and encoding.

(Sixth Embodiment)

Description is now made of a moving picture encoding system according toa sixth embodiment of the present invention.

FIG. 18 is a block diagram showing an example of configuration of themoving picture encoding system according to the sixth embodiment.

Referring to FIG. 18, the moving picture encoding system according tothe sixth embodiment is configured as a specific example of the movingpicture encoding system according to the fourth embodiment shown in FIG.13, including an accumulation controller 412, a second accumulationbuffer 416, a second reference picture buffer 418, and a code ratecontroller 423 added anew.

Description is now made of those component elements in FIG. 18, whichhave different functions or new functions relative to component elementsof the fourth embodiment shown in FIG. 13.

There is a fourth accumulation buffer 501 configured for functions toacquire super-resolution enlarged pictures from a first super-resolutionenlarger 103, to accumulate therein, and supply accumulatedsuper-resolution enlarged pictures to a third predictor 502. The fourthaccumulation buffer 501 may work to acquire information on control foraccumulation from the accumulation controller 412, and operate on theaccumulation control information to control acquisition, accumulation,and supply of super-resolution enlarged pictures.

The third predictor 502 may well be configured for functions to acquirefrom the fourth accumulation buffer 501 super-resolution enlargedpictures, from the second accumulation buffer 418 super-resolutionenlarged decoded pictures, and from a third resolution converter 110resolution conversion enlarged decoded pictures, and have a respectiveone of the super-resolution enlarged pictures thus acquired, as anencoding target picture, a respective one of the resolution conversionenlarged decoded pictures, as a first reference picture from resolutionconversion enlarged decoded picture, and a respective one of thesuper-resolution enlarged decoded pictures, as a second referencepicture from super-resolution enlarged decoded picture, to implementthereon a process for a prescribed prediction to create a predictivepicture, and subtract the predictive picture from the encoding targetpicture, to create data on a difference in between, to supply to a thirdencoder element 503.

The third encoder element 503 is configured for functions to acquiredifference data from the third predictor 502, and implement thereon aprocess for a prescribed encoding to create a third sequence of encodedbits, to supply to a multiplexer 109. The third encoder element 503 maywell be configured for functions to work on control information from thecode rate controller 423, to vary the code rate of the third sequence ofencoded bits being created by the encoding process.

The third resolution converter 110 is configured for functions toacquire decoded pictures from the second accumulation buffer 416, andimplement thereon a process for a prescribed resolution conversion, tocreate resolution conversion enlarged decoded pictures being enlargeddecoded pictures with a high resolution, to supply to the thirdpredictor 502. The third resolution converter 110 may be configured toimplement the process for the prescribed resolution conversion to createresolution conversion enlarged decoded pictures with a high resolutionequal to the spatial resolution of super-resolution enlarged picturescreated at the first super-resolution enlarger 103 and that ofsuper-resolution enlarged decoded pictures created at the secondsuper-resolution enlarger 105.

There will be supplemental description of the accumulation controller412, the second accumulation buffer 416, the second accumulation buffer418, and the code rate controller 423.

The accumulation controller 412 can do with functions equivalent tothose of the accumulation controller 412 in the third embodiment show inFIG. 10, and redundant description is omitted.

The second accumulation buffer 416 can do with functions equivalent tothose of the second accumulation buffer 416 in the third embodiment showin FIG. 10, and redundant description is omitted.

The second accumulation buffer 418 is configured for additionalfunctions to supply super-resolution enlarged decoded pictures acquiredfrom the second super-resolution enlarger 105, to the third predictor502.

Also the code rate controller 423 is configured for functions equivalentto those of the code rate controller 423 in the third embodiment show inFIG. 10.

Such being the case, the sixth embodiment shown in FIG. 18 is adaptedfor operations similar to those of the third embodiment show in FIG. 10,and redundant description is omitted.

The moving picture encoding system according to the sixth embodiment isadapted to exhibit similar advantageous effects to the moving pictureencoding system according to the fourth embodiment shown in FIG. 1, andlike the third embodiment show in FIG. 10, configured with components,such as an accumulation controller 412 and a code rate controller 423,implementing among others accumulation control at respective buffers andcode rate control at a first encoder 102 and a third encoder 108, thuspermitting an encoding to be executed taking into consideration amongothers accumulation amounts at the buffers and code rates at the firstencoder 102 and the third encoder 108.

In the sixth embodiment described, there have been components such as anaccumulation controller 412 and a code rate controller 423 incorporatedin the moving picture encoding system according to the fourth embodimentshown in FIG. 13, to effect accumulation control at respective buffersand code rate control at a first encoder 102 and a third encoder 108,whereas there may well be an example having components such as anaccumulation controller 412 and a code rate controller 423 incorporatedin an example of moving picture encoding system according to the fifthembodiment shown in FIG. 16 or FIG. 17, to likewise effect among othersaccumulation control and code rate control.

(Seventh Embodiment)

Description is now made of a moving picture encoding system according toa seventh embodiment of the present invention.

FIG. 19 is a block diagram showing an example of configuration of themoving picture encoding system according to the seventh embodiment.

As shown in FIG. 19, the moving picture encoding system according to theseventh embodiment is configured as an integration of the moving pictureencoding system according to the first embodiment shown in FIG. 1 andthe moving picture encoding system according to the fourth embodimentshown in FIG. 13, so as for basic functions of component elements,different ones will be described, omitting redundancy.

There is a multiplexer 109 configured for functions to acquire from afirst encoder 102 a first sequence of encoded bits, from a secondencoder 107 a second sequence of encoded bits, and from a third encoder108 a third sequence of encoded bits, and operate complying with aprescribed syntax structure to implement a process of multiplexing thefirst sequence of encoded bits, the second sequence of encoded bits, andthe third sequence of encoded bits, together with sequences of encodedbits of sets of data used in encoding processes, involving data onmotion vectors, data on quantizing parameters, data on reference pictureselection, and data on encoding parameters, as they are eachrespectively processed through an unshown entropy encoder, whileinserting identification data for identification of a set of subsequentsequences of encoded bits and the like, to create a sequence ofmultiplexed bits. The multiplexer 109 may then be configured toreproduce a first sequence of encoded bits acquired from the firstencoder 102 with a standard resolution, to have a first sequence of bitsencoded by a first encoding, and a second sequence of bits encoded bythe first encoding, and operate complying with a prescribed syntaxstructure to implement a process of multiplexing the first sequence ofbits encoded by the first encoding as reproduced with the standardresolution, and the second sequence of encoded bits as acquired,together with sequences of encoded bits of sets of data used in encodingprocesses, involving data on motion vectors, data on quantizingparameters, data on reference picture selection, and data on encodingparameters, as they are each respectively processed through an unshownentropy encoder, while inserting identification data for identificationof a set of subsequent sequences of encoded bits, to create a sequenceof encoded bits as multiplexed to output. The multiplexer 109 may beconfigured to operate complying with a prescribed syntax structure toimplement a process of multiplexing the second sequence of bits encodedby the first encoding as reproduced, and the third sequence of encodedbits as acquired, together with sequences of encoded bits of sets ofdata used in encoding processes, involving data on motion vectors, dataon quantizing parameters, data on reference picture selection, and dataon encoding parameters, as they are each respectively processed throughan unshown entropy encoder, while inserting identification data foridentification of a set of subsequent sequences of encoded bits and thelike, to create a second sequence of encoded bits as multiplexed tooutput.

Further, the multiplexer 109 may be configured to acquire from thesecond encoder 107 a first to a third sequence of bits encoded by asecond encoding, and from the third encoder 108 a first to a thirdsequence of bits encoded by a third encoding. The multiplexer 109 maythen be preferably configured for functions to reproduce a firstsequence of encoded bits acquired with a standard resolution, to have afirst to a sixth sequence of bits encoded by a first encoding, andoperate complying with a prescribed syntax structure to implement aprocess of multiplexing a respective one of a combination of the firstsequence of bits encoded by the second encoding and the first sequenceof bits encoded by the first encoding with the standard resolution, acombination of a second sequence of bits encoded by the second encodingand a second sequence of bits encoded by the first encoding with thestandard resolution, a combination of the third sequence of bits encodedby the second encoding and a third sequence of bits encoded by the firstencoding with the standard resolution, a combination of the firstsequence of bits encoded by the third encoding and a fourth sequence ofbits encoded by the first encoding with the standard resolution, acombination of a second sequence of bits encoded by the third encodingand a fifth sequence of bits encoded by the first encoding with thestandard resolution, and a combination of the third sequence of bitsencoded by the third encoding and the sixth sequence of bits encoded bythe first encoding with the standard resolution, and sequences ofencoded bits of sets of data used in encoding processes, involving dataon motion vectors, data on quantizing parameters, data on referencepicture selection, and data on encoding parameters, as they are eachrespectively processed through an unshown entropy encoder, whileinserting identification data for identification of a set of subsequentsequences of encoded bits and the like, to create a first to a sixthsequence of encoded bits as multiplexed to output as requested. Themultiplexer 109 may then be configured to operate in response to ademand, to selectively create any of the first to the sixth sequence ofencoded bits as multiplexed.

With functions inclusive of the foregoing, the seventh embodiment isconfigured to work on input moving pictures, to implement a process fora prescribed super-resolution enlargement and a process for a prescribedresolution conversion, to create among others super-resolution enlargedsignals and super-resolution enlarged and converted signals of inputpictures, to make use of them allowing for a moving picture encodingbased on an increased amount of information relative to an informationamount of input moving pictures. In the configuration shown in FIG. 19,there is combination of a first super-resolution enlarger 103 and asecond super-resolution enlarger 105 that have enlargement ratiosapplied thereto, which may well be equal to each other, while they maybe unequal to each other. Moreover, there is combination of a firstresolution converter 104 and a second resolution converter 106 that haveresolution conversion ratios applied thereto, which may well be such anidentical resolution conversion ratio that would provide a standardresolution as a spatial resolution after the process for resolutionconversion, while they may be different enlargement ratios. Further,there is a third resolution converter 110 that has a resolutionconversion ratio applied thereto, which may well be such a resolutionconversion ratio that would provide the same resolution as the spatialresolution of super-resolution enlarged pictures created at the firstsuper-resolution enlarger 103 and super-resolution enlarged decodedpictures created at the second super-resolution enlarger 105, while itmay be a different enlargement ratio.

Description is now made of actions of the moving picture encoding systemaccording to the seventh embodiment shown in FIG. 19, with reference toa flowchart of FIG. 20.

First, there are input moving pictures stored in a first accumulationbuffer 101, as necessary in number of pictures for the firstsuper-resolution enlarger 103 to implement a process for a firstsuper-resolution enlargement (S1901).

The first super-resolution enlarger 103 acquires from the firstaccumulation buffer 101 two or more pictures, as necessary for a processfor a prescribed first super-resolution enlargement, and implementsthereon the process for the prescribed super-resolution enlargement(S1902), whereby it creates super-resolution enlarged pictures with aresolution higher than a standard resolution that input moving pictureshave as a spatial resolution thereof, to supply to the first resolutionconverter 104 and the third encoder 108.

After that, the third encoder 108 acquires super-resolution enlargedpictures from the first super-resolution enlarger 103, storing them in aprescribed buffer for temporary accumulation (S1903).

After that, the first resolution converter 104 acquires super-resolutionenlarged pictures from the first super-resolution enlarger 103, andimplements thereon a process for a prescribed resolution conversion(S1904), whereby it creates super-resolution enlarged and convertedsignals of input pictures with the standard resolution, to supply thuscreated super-resolution enlarged and converted signals of inputpictures with the standard resolution to the second encoder 107.

After that, the second encoder 107 acquires from the first resolutionconverter 104 super-resolution enlarged and converted signals of inputpictures with the standard resolution, storing them in a prescribedbuffer for temporary accumulation (S1905).

On the other hand, the first encoder 102 acquires from the firstaccumulation buffer 101 a sequence of moving pictures with the standardresolution as necessary for implementation of a process for a prescribedpre-super-resolution encoding, and implements thereon a process for anencoding and a decoding at the standard resolution (S1906), whereby itcreates a first sequence of encoded bits with the standard resolution asa result of the encoding process, and a set of decoded pictures with thestandard resolution as a result of the decoding. After that, the firstencoder 102 works to supply the first sequence of encoded bits thuscreated to the multiplexer 109, and supply the set of decoded picturescreated with the standard resolution to the second super-resolutionenlarger 105, the second encoder 107, and the third resolution converter110.

Then, the second encoder 107 acquires the set of decoded pictures fromthe first encoder 102, storing in a prescribed buffer for temporaryaccumulation (S1907).

Further, the second super-resolution enlarger 105 acquires the set ofdecoded pictures from the first encoder 102, and implements thereon aprocess for a prescribed super-resolution enlargement (S1908), wherebyit creates a set of super-resolution enlarged decoded pictures with aresolution higher than the standard resolution, to supply to the secondresolution converter 106 and the third encoder 108.

Then, the third encoder 108 acquires the set of super-resolutionenlarged decoded pictures from the second super-resolution enlarger 105,storing in a prescribed buffer for temporary accumulation (S1909).

Further, the second resolution converter 106 acquires the set ofsuper-resolution enlarged decoded pictures from the secondsuper-resolution enlarger 105, and implements thereon a process for aprescribed resolution conversion (S1910), whereby it createssuper-resolution enlarged and converted decoded signals with thestandard resolution, to supply to the second encoder 107.

Then, the second encoder 107 acquires super-resolution enlarged andconverted decoded signals with the standard resolution from the secondresolution converter 106, storing them in a prescribed buffer fortemporary accumulation (S1911).

Further, the third resolution converter 110 acquires the set decodedpictures from the first encoder 102, and implements thereon a processfor a prescribed resolution conversion (S1912), whereby it creates a setof resolution conversion enlarged decoded pictures being enlargeddecoded pictures with a high resolution, to supply to the third encoder108.

After that, the third encoder 108 acquires the set of resolutionconversion enlarged decoded pictures with the high resolution from thethird resolution converter 110, storing in a prescribed buffer fortemporary accumulation (S1913).

The combination of processes associated with steps S1902 to S1905, thecombination of processes associated with steps S1906 to S1911, and thecombination of processes associated with steps S1912 to S1913 may beimplemented in parallel, or in series.

With the foregoing processes completed, the second encoder 107 hassuper-resolution enlarged and converted pictures of input picturesacquired from the first resolution converter 104 as encoding targetpictures, decoded pictures acquired from the first encoder 102 as firstreference pictures, and super-resolution enlarged and converted decodedpictures acquired from the second resolution converter 106 as secondreference pictures, and works to implement thereon a process for aprescribed second prediction to create data on differences at thestandard resolution (S1914), implement a second encoding process as aprocess for a prescribed second encoding on the difference data thuscreated (S1915), whereby it creates a second sequence of encoded bits,to supply to the multiplexer 109.

Further, the third encoder 108 has super-resolution enlarged pictures ofinput pictures acquired from the first super-resolution enlarger 103 asencoding target pictures, resolution conversion enlarged decodedpictures acquired from the third resolution converter 110 as firstreference pictures, and super-resolution enlarged decoded picturesacquired from the second super-resolution enlarger 105 as secondreference pictures, and works to implement thereon a process for aprescribed third prediction to create data on differences at the highresolution (S1916), and implement a third encoding process being aprescribed third encoding on the data on differences thus created to(S1917), whereby it creates a third sequence of encoded bits, to supplyto the multiplexer 109. After that, it waits for completion of otherparallel processes.

After that, the multiplexer 109 acquires a first sequence of encodedbits from the first encoder 102, a second sequence of encoded bits fromthe second encoder 107, and a third sequence of encoded bits from thethird encoder 108, and works complying with a prescribed syntaxstructure, to implement a process of multiplexing the first sequence ofencoded bits, the second sequence of encoded bits, and the thirdsequence of encoded bits, together with sets of data used in encodingprocesses, involving data on motion vectors, data on quantizingparameters, data on reference picture selection, and data on encodingparameters, as they area each respectively processed through an unshownentropy encoder, while inserting identification data for identificationof a set of subsequent sequences of encoded bits and the like (S1918),to create a sequence of multiplexed bits. The present embodimentinvolves a series of actions to be complete through the foregoing steps.

FIG. 21 is a diagram of data format illustrating an example of structureof a multiplexed bit sequence 300 created at the multiplexer 109according to the seventh embodiment.

The multiplexed bit sequence 300 has multiplexed therein a bit sequence310 of data on encoding parameters, a first sequence 320 of encoded bitsfrom the first encoder 102, a second sequence 330 of encoded bits fromthe second encoder 107, and a third sequence 340 of encoded bits fromthe third encoder 108.

There have been operations described as being parallel processesaccording to the present embodiment, while those processed in parallelmay well be consecutively processed in a configuration operableaccording to the present embodiment.

Therefore, the seventh embodiment, configured as shown in FIG. 19,affords to make simultaneous use of advantageous effects available fromthe first embodiment shown in FIG. 1 and those from the fourthembodiment shown in FIG. 12.

Further, according to the seventh embodiment, there is a systemincluding a first encoder 102 configured to work on moving picturesinput with a standard resolution, to implement thereon a combination ofprocesses for prescribed encoding and decoding to create a firstsequence of encoded bits with the standard resolution, a second encoder107 configured to implement a process for a second encoding to create asecond sequence of encoded bits with the standard resolution, and athird encoder 108 configured to implement a process for a third encodingto create a third sequence of encoded bits with a high resolution, thesystem being adapted to implement a process for a prescribedmultiplexing on the first to the third sequence of encoded bits tocreate a sequence of multiplexed bits, as a single stream of encodedbits carrying results of encoding on inherent input moving pictures, andresults of encoding on a set of errors in a prediction using informationon frequency components as enlarged in among others super-resolutionenlarged pictures and super-resolution enlarged and converted picturescreated by a process for super-resolution enlargement combined with aprocess for resolution conversion, permitting their efficientaccumulation or transmission, allowing for a selective decoding at adecoding system end.

Further, according to the seventh embodiment, there is a systemconfigured with a second encoder 107 adapted to implement a process fora prescribed prediction making use of a first reference picture obtainedfrom a decoded picture with a standard resolution or a second referencepicture obtained from a set of super-resolution enlarged and convertedsignals of the decoded picture with the standard resolution, whicheveris used as a reference picture, to create a predictive picture, andsubtract the predictive picture from an encoding target picture obtainedfrom a set of super-resolution enlarged and converted signals of aninput picture with the standard resolution, to create a data on adifference in between, implementing a process for a prescribed encodingon the difference data to create a second sequence of encoded bits, incombination with a third encoder 108 adapted to implement a process fora prescribed prediction making use of a first reference picture obtainedfrom a resolution conversion enlarged decoded picture with a highresolution or a second reference picture obtained from asuper-resolution enlarged decoded picture, whichever is used as areference picture, to create a predictive picture, and subtract thispredictive picture from an encoding target picture obtained from asuper-resolution enlarged picture, to create a data on a difference inbetween, implementing a process for a prescribed encoding on thisdifference data to create a third sequence of encoded bits, therebyaffording to operate in accordance with a request, to perform ahierarchical encoding making use of a correlation of spatial resolutionbetween the decoded picture and the set of super-resolution enlarged andconverted signals of the input picture with the standard resolution, aswell as in use of a correlation of spatial resolution between the set ofsuper-resolution enlarged and converted decoded signals and the set ofsuper-resolution enlarged and converted signals of the input picturewith the standard resolution, and perform a hierarchical encoding makinguse of a correlation of spatial resolution between the super-resolutionenlarged picture and the resolution conversion enlarged decoded picture,as well as in use of a correlation of spatial resolution between thesuper-resolution enlarged picture and the super-resolution enlargeddecoded picture, thus allowing for different multiplexed bit sequencesto be created to supply to, accumulate at, and/or transmit to a decodingsystem end.

Moreover, it is possible to provide a configuration for services tocreate information on reference picture selection including a data onreference picture selection for identifying a first reference picture ora second reference picture whichever is used, to supply to themultiplexer 109, allowing for a facilitated identification of areference picture used in an encoding according to the seventhembodiment.

Further, according to the seventh embodiment, there is a system adaptedto operate in accordance with a request, to implement a process for amultiplexing on a domain including a first sequence of encoded bits asderived from a first encoder 102 with a standard resolution, a first toa third sequence of bits encoded by a second encoding as derived from asecond encoder 107, and a first to a third sequence of bits encoded by athird encoding as derived from a third encoder 107, to create any offirst to sixth sequences of encoded bits as multiplexed, permittingcreation of a variety of encoded bit sequences. Further, there is asystem adaptive for services such as transmitting to an unshown externalaccumulator before recording in a prescribed recording medium, ordelivering through a network using an unshown external communicationsystem, thereby permitting a moving picture encoding system according tothe present embodiment to operate on moving pictures with heavy amountsof information, allowing for efficient encoding, accumulation, andtransmission, as well as for selective decoding at a decoding systemend.

In FIG. 19, the moving picture encoding system according to the seventhembodiment is configured as an integration of the moving pictureencoding system according to the first embodiment shown in FIG. 1 andthe moving picture encoding system according to the fourth embodimentshown in FIG. 13, while there may well be integrations such as anintegration of the moving picture encoding system according to the firstembodiment shown in FIG. 1 and an example of moving picture encodingsystem according to the fifth embodiment shown in FIG. 16 or FIG. 17, anintegration of an example of moving picture encoding system according tothe second embodiment shown in FIG. 7, FIG. 8, or FIG. 9 and an exampleof moving picture encoding system according to the fifth embodimentshown in FIG. 16 or FIG. 17, and an integration of an example of movingpicture encoding system according to the second embodiment shown in FIG.7, FIG. 8, or FIG. 9 and the moving picture encoding system according tothe fourth embodiment shown in FIG. 13.

(Eighth Embodiment)

Description is now made of a moving picture encoding system according toan eighth embodiment of the present invention.

FIG. 22 is a block diagram showing an example of configuration of themoving picture encoding system according to the eighth embodiment.

Referring to FIG. 22, the moving picture encoding system according tothe eighth embodiment is comprised of a set of component elementsequivalent to an integration of the fourth embodiment shown in FIG. 13and the sixth embodiment shown in FIG. 18. Accordingly, the eighthembodiment is adapted for a set of operations equivalent to acombination of operations of the fourth embodiment and operations of thesixth embodiment shown, so redundant description is omitted.

The moving picture encoding system according to the eighth embodiment isadapted to exhibit similar advantageous effects to the moving pictureencoding system according to the seventh embodiment shown in FIG. 19,and like the third embodiment show in FIG. 10, the sixth embodiment showin FIG. 18, or the like, configured with components, such as anaccumulation controller 412 and a code rate controller 423, implementingamong others accumulation control at respective buffers and code ratecontrol at a first encoder 102 and a third encoder 108, thus permittingan encoding to be executed taking into consideration among othersaccumulation amounts at the buffers and code rates at the first encoder102 and the third encoder 108.

In the first to the eighth embodiment described, there have been movingpicture encoding systems depicted by block diagrams and described ashardware configurations, whereas according to the present inventionbeing not limited thereto, there may well be moving picture encodingprograms stored in CPU's and among others memories and HDD's, to attainfunctions of the first to the eighth embodiment by software processes,respectively. According to the present invention, there are movingpicture encoding systems, moving picture encoding methods, and movingpicture encoding programs having their ranges of application to suchapparatuses or systems, methods, and programs as adapted for encodingmoving pictures, without specific limitations. For the presentinvention, applications cited may involve, for instance, broadcastingequipment typified by TV, mobile telephones, teleconferences, monitors,DVD-R/RW or BD-R/RW, HDD, SD®, recording and reproducing systems using arecordable and rewritable recording medium such as a holographic memory,imaging, recording and reproducing systems such as a camcorder,recording and editing systems such as a an authoring system, anddelivering systems for moving pictures.

(Ninth Embodiment)

There will be described moving picture decoding systems embodiedaccording to the present invention, as a ninth embodiment et seq.

FIG. 23 is a block diagram showing an example of configuration of amoving picture decoding system according to the ninth embodiment.

Referring to FIG. 23, the moving picture decoding system according tothe ninth embodiment includes a demultiplexer 1801, a first decoder1802, a first super-resolution enlarger 1803, a first resolutionconverter 1804, a second decoder 1805, a second resolution converter1806, and a third decoder 1807.

The demultiplexer 1801 is configured for functions to acquire as aninput a sequence of multiplexed encoded bits input thereto, and workcomplying with a prescribed syntax structure, to make a demultiplexing,while identifying identification data for identification of sets of datainvolving data on encoding modes and data on parameters of respectivetypes as they are used in encoding processes, to acquire from thesequence of multiplexed encoded bits a first sequence of encoded bits tobe decoded at the first decoder 1802, a second sequence of encoded bitsto be decoded at the second decoder 1805, and a third sequence ofencoded bits to be decoded at the third decoder 1807. The demultiplexer1801 may well be configured for functions to additionally acquire, ifavailable any from the sequence of multiplexed encoded bits, sequencesof bits of data on enlargement ratios, sequences of bits of data onresolution conversion ratios, sequences of encoded bits of data onselection of reference pictures and data on selection modes of referencepictures to be used in a process for a second decoding, and sequences ofencoded bits of data on selection of reference pictures and data onselection modes of reference pictures to be used in a process for athird decoding, to supply as necessary. The demultiplexer 1801 isconfigured for functions at least to supply the first decoder 1802 withthe first sequence of encoded bits, the second decoder 1805 with thesecond sequence of encoded bits, and the third decoder 1807 with thethird sequence of encoded bits, as they are acquired. The demultiplexer1801 may well be configured for functions to acquire, as possible if anyfrom the sequence of multiplexed encoded bits, sequences of encoded bitsof data on motion vectors, to supply to the first decoder 1802.

The first decoder 1802 is configured for functions to acquire a firstsequence of encoded bits with a standard resolution from thedemultiplexer 1801, and implement thereon a process for a prescribedfirst decoding, to create a sequence of decoded pictures with thestandard resolution. The first decoder 1802 may well be configured forfunctions to acquire, as possible if any through an unshown entropydecoder, a set of data on motion vectors obtained by implementation of aprocess for a prescribed entropy decoding, for use to implement theprocess for the prescribed decoding. The first decoder 1802 isconfigured for functions to supply the sequence of decoded picturescreated with the standard resolution to the first super-resolutionenlarger 1803, the second decoder 1805, and the second resolutionconverter 1806. The first decoder 1802 may well be configured forfunctions to temporarily store the sequence of decoded pictures. Thefirst decoder 1802 may well be configured for functions to work inaccordance with a request, to supply the sequence of decoded picturescreated with the standard resolution to an unshown external displaysystem, affording to provide an output for display of sequences ofdecoded pictures with the standard resolution.

The first super-resolution enlarger 1803 is configured for functions toacquire from the first decoder 1802 a sequence of decoded pictures withthe standard resolution, for acquisition of two or more decoded pictureswith the standard resolution as necessary for a process for a prescribedsuper-resolution enlargement, work on the sequence of decoded picturesacquired with the standard resolution to implement the process for theprescribed super-resolution enlargement, to create and output a sequenceof super-resolution enlarged decoded pictures with a resolution higherthan the standard resolution. For the process for the prescribedsuper-resolution enlargement to be given an enlargement ratio, there maywell be a configuration for functions to acquire data on the enlargementratio as it is preset as a prescribed enlargement ratio, or obtained asa data acquired at the demultiplexer 1801 and processed for a prescribedentropy decoding at an unshown entropy decoder, and work on the data onenlargement ratio thus acquired to establish an enlargement ratio, foruse to implement the process for the prescribed super-resolutionenlargement. The first super-resolution enlarger 1803 is configured forfunctions to supply the sequence of super-resolution enlarged decodedpictures thus created, to the first resolution converter 1804. The firstsuper-resolution enlarger 1803 may well be configured for functions towork in accordance with a request, to supply the sequence ofsuper-resolution enlarged decoded pictures created to an unshownexternal display system, affording to provide an output for display ofsequences of super-resolution enlarged decoded pictures with theresolution higher than the standard resolution.

The first resolution converter 1804 is configured for functions toacquire from the first super-resolution enlarger 1803 a sequence ofsuper-resolution enlarged decoded pictures, and work on the sequence ofsuper-resolution enlarged decoded pictures as acquired, to implement aprocess for a prescribed resolution conversion to create, from thesequence of super-resolution enlarged decoded pictures with theresolution higher than the standard resolution, a sequence ofsuper-resolution decoded pictures with the standard resolution, tosupply to the second decoder 1805. For the process for the prescribedresolution conversion to be given a resolution conversion ratio, theremay well be a configuration for functions to acquire data on theresolution conversion ratio as is it is preset as a prescribedresolution conversion ratio or, if possible, obtained as a data from thedemultiplexer 1801 as it is processed for a prescribed entropy decodingat an unshown entropy decoder, and work on the data on resolutionconversion ratio thus acquired to establish a resolution conversionratio, for use to implement the process for the prescribed resolutionconversion. The first resolution converter 1804 may well be configuredfor functions to work in accordance with a request, to supply thesequence of super-resolution decoded pictures created to an unshownexternal display system, affording to provide an output for display ofsequences of super-resolution decoded pictures with the standardresolution.

The second decoder 1805 is configured for functions to acquire a secondsequence of encoded bits with the standard resolution from thedemultiplexer 1801, a sequence of decoded pictures with the standardresolution from the first decoder 1802, and a sequence ofsuper-resolution decoded pictures with the standard resolution from thefirst resolution converter 1804, and work on the second sequence ofencoded bits as acquired, to implement a process for a prescribed seconddecoding being an extension decoding at the standard resolution, tocreate a set of decoded difference data at the standard resolution. Thesecond decoder 1805 may well be configured for functions to temporarilystore the sequences of pictures as acquired.

The second decoder 1805 is configured for functions to have a respectivedecoded picture in the decoded picture sequence, as a first referencepicture, and a respective super-resolution decoded picture in thesuper-resolution decoded picture sequence, as a second referencepicture, for use to implement a process for a prescribed prediction tocreate predictive pictures with the standard resolution. The seconddecoder 1805 may be configured better with functions to acquire, ifpossible, data on reference picture selection and data on referencepicture selection modes obtained from the demultiplexer 1801 as they areprocessed for a prescribed entropy decoding at an unshown entropydecoder, and work on the data on reference picture selection and thedata on reference picture selection modes, to implement the process forthe prescribed prediction to create predictive pictures with thestandard resolution. For the process for the prescribed prediction,there may well be a configuration to follow the data on referencepicture selection and the data on reference picture selection modes asacquired, to define what reference pictures were used under whatencoding modes to prepare predictive pictures in creation of the secondsequence of encoded bits constituting decoding targets, and work so asto make predictive pictures equivalent to those in the encoding. Forinstance, if it is defined from the data on reference picture selectionmodes that employed to implement a prediction process was an encodingmode using either a set of first reference pictures or a set of secondreference pictures, then the second decoder 1805 is to select a set ofreference pictures thus identified, for use to implement the process forthe prescribed prediction to create predictive pictures. Or forinstance, if it is defined from the data on reference picture selectionmodes that employed to implement a prediction process was an encodingmode using neither a set of first reference pictures nor a set of secondreference pictures, then the second decoder 1805 is to implement theprocess for the prescribed prediction to create predictive pictures,using neither of the sets of reference pictures. Or for instance, if itis defined from the data on reference picture selection modes thatemployed to implement a prediction process was an encoding modecomprised of controlling selection of reference picture for each pictureor for each set of a prescribed number of pictures, then the seconddecoder 1805 is to make use of the data on reference picture selection,to identify a first reference picture or a second reference picture,whichever is selective in order, to use reference pictures thusidentified to implement the process for the prescribed prediction tocreate predictive pictures. Or for instance, if it is defined from thedata on reference picture selection modes that employed to implement aprediction process was an encoding mode comprised of having a respectivetarget picture divided with no spaces left into regions of a prescribedarea and also a first reference picture and a second reference pictureeach likewise divided with no spaces left into regions of a prescribedarea, and making a control for each combination of corresponding regionsto select the first or the second reference picture for use, then thesecond decoder 1805 is to make use of the data on reference pictureselection, to identify for each combination of corresponding regions afirst reference picture or a second reference picture, whichever isselective in order, to use reference pictures thus identified toimplement the process for the prescribed prediction to create predictivepictures.

The second decoder 1805 is configured for functions to add decodeddifference data created at the standard resolution to predictivepictures with the standard resolution, to create a sequence ofsuper-resolution pictures as decoded with the standard resolution. Thesecond decoder 1805 may well be configured to work in accordance with arequest, to supply the created sequence of super-resolution pictures asdecoded with the standard resolution to an unshown external displaysystem, affording to provide an output for display of sequences ofsuper-resolution pictures with the standard resolution.

The second resolution converter 1806 is configured for functions toacquire from the first decoder 1802 a sequence of decoded pictures withthe standard resolution, and work on the sequence of decoded picturesacquired with the standard resolution, to implement a process for aprescribed resolution conversion, to create a sequence of enlargeddecoded pictures with a high resolution having the same resolution asthe spatial resolution of the sequence of super-resolution enlargeddecoded pictures created at the first super-resolution enlarger 1803, tosupply to the third decoder 1807. For the process for the prescribedresolution conversion to be given resolution a conversion ratio, theremay well be a configuration for functions to acquire data on theresolution conversion ratio as it is preset as a prescribed resolutionconversion ratio or, if possible, obtained as a data from thedemultiplexer 1801 as it is processed for a prescribed entropy decodingat an unshown entropy decoder, and work on the data on the resolutionconversion ratio thus acquired to establish a resolution conversionratio, for use to implement the process for the prescribed resolutionconversion. The second resolution converter 1806 may well be configuredfor functions to work in accordance with a request, to supply thesequence of super-resolution decoded pictures created to an unshownexternal display system, affording to provide an output for display ofsequences of super-resolution decoded pictures with the standardresolution.

The third decoder 1807 is configured for functions to acquire a thirdsequence of encoded bits with the high resolution from the demultiplexer1801, a sequence of enlarged decoded pictures with the high resolutionfrom the second resolution converter 1806, and a sequence ofsuper-resolution enlarged decoded pictures with the high resolution fromthe first super-resolution enlarger 1803, and work on the third sequenceof encoded bits as acquired, to implement a process for a prescribedthird decoding being an extension decoding at the high resolution, tocreate a set of decoded difference data at the high resolution. Thethird decoder 1807 may well be configured for functions to temporarilystore the sequences of pictures as acquired.

The third decoder 1807 is configured for functions to have a respectiveenlarged decoded picture in the enlarged decoded picture sequence, as afirst reference picture, and a respective super-resolution enlargeddecoded picture in the super-resolution enlarged decoded picturesequence, as a second reference picture, for use to implement a processfor a prescribed prediction to create predictive pictures with the highresolution. The third decoder 1807 may be configured better withfunctions to acquire, if possible, data on reference picture selectionand data on reference picture selection modes obtained from thedemultiplexer 1801 as they are processed for a prescribed entropydecoding at an unshown entropy decoder, and work on the data onreference picture selection and the data on reference picture selectionmodes, to implement the process for the prescribed prediction to createpredictive pictures with the high resolution. For the process for theprescribed prediction, there may well be a configuration to follow thedata on reference picture selection and the data on reference pictureselection modes as acquired, to define what reference pictures were usedunder what encoding modes to prepare predictive pictures in creation ofthe third sequence of encoded bits constituting decoding targets, andwork so as to make predictive pictures equivalent to those in theencoding. For instance, if it is defined from the data on referencepicture selection modes that employed to implement a prediction processwas an encoding mode using either a set of first reference pictures or aset of second reference pictures, then the third decoder 1807 is toselect a set of reference pictures thus identified, for use to implementthe process for the prescribed prediction to create predictive pictures.Or for instance, if it is defined from the data on reference pictureselection modes that employed to implement a prediction process was anencoding mode using neither a set of first reference pictures nor a setof second reference pictures, then the third decoder 1807 is toimplement the process for the prescribed prediction to create predictivepictures, using neither of the sets of reference pictures. Or forinstance, if it is defined from the data on reference picture selectionmodes that employed to implement a prediction process was an encodingmode comprised of controlling selection of reference picture for eachpicture or for each set of a prescribed number of pictures, then thethird decoder 1807 is to make use of the data on reference pictureselection, to identify a first reference picture or a second referencepicture, whichever is selective in order, to use reference pictures thusidentified to implement the process for the prescribed prediction tocreate predictive pictures. Or for instance, if it is defined from thedata on reference picture selection modes that employed to implement aprediction process was an encoding mode comprised of having a respectivetarget picture divided with no spaces left into regions of a prescribedarea and also a first reference picture and a second reference pictureeach likewise divided with no spaces left into regions of a prescribedarea, and making a control for each combination of corresponding regionsto select the first or the second reference picture for use, then thethird decoder 1807 is to make use of the data on reference pictureselection, to identify for each combination of corresponding regions afirst reference picture or a second reference picture, whichever isselective in order, to use reference pictures thus identified toimplement the process for the prescribed prediction to create predictivepictures.

The third decoder 1807 is configured for functions to add decodeddifference data created at the high resolution to predictive pictureswith the high resolution, to create a sequence of super-resolutionenlarged pictures as decoded with the high resolution. The third decoder1807 may well be configured to work in accordance with a request, tosupply the created sequence of super-resolution enlarged pictures asdecoded with the high resolution to an unshown external display systemfor instance, affording to provide an output for display of sequences ofsuper-resolution enlarged pictures with the high resolution.

The moving picture decoding system configured as described is adapted toinput multiplexed sequences of encoded bits created at a moving pictureencoding system according to any of the first to the eighth embodimentor the like, and decode in reverse to the encoding, affording to acquiredecoded picture sequences with more diverse spatial resolutions orfrequency components and higher qualities than ever, such as decodedpicture sequences with standard resolution, enlarged decoded picturesequences, super-resolution enlarged decoded picture sequences,super-resolution decoded picture sequences, super-resolution picturesequences as decoded, and super-resolution enlarged picture sequences asdecoded. It also is possible to provide a configuration adapted tooperate in accordance with a request for a selective output of suchdiverse decoded picture sequences to an unshown display system. Forinstance, for users of those display devices non-adaptive to highresolution display, such as mobile terminal display devices, it ispossible to request selective one of a decoded picture sequence withstandard resolution, a super-resolution decoded picture sequence withstandard resolution, and a super-resolution picture sequence as decodedwith standard resolution, as an output to view and listen. Or forinstance, for users of those display devices adaptive to high resolutiondisplay, such as high-definition televisions, it is possible to requestselective one of an enlarged decoded picture sequence with highresolution, a super-resolution enlarged decoded picture sequence withhigh resolution, and a super-resolution enlarged picture sequence asdecoded, as an output to view and listen.

Description is now made of actions of the moving picture decoding systemaccording to the ninth embodiment shown in FIG. 23, with reference to aflowchart of FIG. 24.

First, the demultiplexer 1801 acquires a sequence of multiplexed bitsinput thereto, and works thereon complying with a prescribed syntaxstructure, to make a demultiplexing (step S601) while identifyingidentification data for identification of among others data on encodingmodes and data on parameters of respective types used in encodingprocesses, to acquire a first sequence of encoded bits with the standardresolution, a second sequence of encoded bits with the standardresolution, and a third sequence of encoded bits with the highresolution, and supply the first sequence of encoded bits to the firstdecoder 1802, the second sequence of encoded bits to the second decoder1805, and the third sequence of encoded bits to third decoder 1807.Further, the demultiplexer 1801 acquires data on motion vectors ifavailable from the sequence of multiplexed bits, and supplies to thefirst decoder 1802.

After that, the flow branches into a first parallel process (steps S602and S603), a second parallel process (step S604), and a third parallelprocess (step S605), to enter parallel actions.

In the first parallel process, first, the first decoder 1802 acquiresfrom the demultiplexer 1801 the first sequence of encoded bits and, ifavailable, data on motion vectors, and implements thereon a process fora first decoding at the standard resolution (step S602), to create asequence of decoded pictures with the standard resolution, and supply tothe second decoder 1805.

Then, the second decoder 1805 acquires the sequence of decoded picturescreated with the standard resolution at the first decoder 1802, andstores in a prescribed buffer for temporary accumulation (step S603).After that, it waits for completion of other parallel processes.

In the second parallel process, the second decoder 1805 acquires thesecond sequence of encoded bits with the standard resolution from thedemultiplexer 1801, and implements thereon a process for a prescribedsecond decoding (step S604), to create a set of decoded difference dataat the standard resolution. After that, it waits for completion of otherparallel processes.

In the third parallel process, the third decoder 1807 acquires the thirdsequence of encoded bits with the high resolution from the demultiplexer1801, and implements thereon a process for a prescribed third decoding(step S605), to create a set of decoded difference data at the highresolution. After that, it waits for completion of other parallelprocesses.

With the first to the third parallel process completed, the flowbranches into a fourth parallel process (steps S606, S607, S608, andS609), and a fifth parallel process (steps S610 and S611), to enterparallel actions.

In the fourth parallel process, the first super-resolution enlarger 1803acquires the sequence of decoded pictures with the standard resolutionfrom the first decoder 1802, and implements thereon a process for aprescribed super-resolution enlargement (step S606), to create asequence of super-resolution enlarged decoded pictures with the highresolution being a resolution higher than the standard resolution, andsupply to the first resolution converter 1804 and the third decoder1807.

Then, the third decoder 1807 acquires the sequence of super-resolutionenlarged decoded pictures created with the high resolution at the firstsuper-resolution enlarger 1803, and stores in a prescribed buffer fortemporary accumulation (step S607).

Then, the first resolution converter 1804 acquires the sequence ofsuper-resolution enlarged decoded pictures with the high resolution fromthe first super-resolution enlarger 1803, and implements thereon aprocess for a prescribed resolution conversion (step S608) to create,from the sequence of super-resolution enlarged decoded pictures with theresolution higher than the standard resolution, a sequence ofsuper-resolution decoded pictures with the standard resolution, andsupply to the second decoder 1805.

Then, the second decoder 1805 acquires the sequence of super-resolutiondecoded pictures created with the standard resolution at the firstresolution converter 1804, and stores in a prescribed buffer fortemporary accumulation (step S609). After that, it waits for completionof the other parallel process.

In the fifth parallel process, the second resolution converter 1806acquires the sequence of decoded pictures with the standard resolutionfrom the first decoder 1802, and implements thereon a process for aprescribed resolution conversion (step S610), to create a sequence ofenlarged decoded pictures with the high resolution, and supply to thethird decoder 1807.

Then, the third decoder 1807 acquires the sequence of enlarged decodedpictures created with the high resolution at the second resolutionconverter 1806, and stores in a prescribed buffer for temporaryaccumulation (step S611). After that, it waits for completion of theother parallel process.

With the foregoing parallel processes completed, the flow branches intoa sixth parallel process (step S612) and a seventh parallel process(step S613), to enter parallel actions.

In the sixth parallel process, the second decoder 1805 implements aprocess for a prescribed prediction (step S612) to create a set ofpredictive pictures with the standard resolution, and makes an additionof created predictive pictures with the standard resolution and decodeddifference data at the standard resolution, to create a sequence ofsuper-resolution pictures as decoded with the standard resolution. Afterthat, it waits for completion of the other parallel process.

In the seventh parallel process, the third decoder 1807 implements aprocess for a prescribed prediction (step S613) to create a set ofpredictive pictures with the high resolution, and makes an addition ofcreated predictive pictures with the high resolution and decodeddifference data at the high resolution, to create a sequence ofsuper-resolution enlarged pictures as decoded with the high resolution.After that, it waits for completion of the other parallel process.

With the parallel processes completed, there is a set of sequences ofdecoded pictures created with described resolutions at the first decoder1802, the first super-resolution enlarger 1803, the first resolutionconverter 1804, the second decoder 1805, the second resolution converter1806, and the third decoder 1807 each operable in response to a requestfor supply to an unshown external display system or the like, to providean output for display of a variety of decoding results on demand (stepS614). The present embodiment involves a series of actions to becomplete through the foregoing steps.

According to the ninth embodiment, there is a moving picture decodingsystem operable through execution of such the steps to create sequencesof decoded pictures as necessary.

It is noted that there have been operations described as being parallelprocesses according to the ninth embodiment, while those processed inparallel may well be consecutively processed in a configuration operableaccording to the present embodiment.

FIG. 25 is a flowchart showing an example of operational procedure ofthe process for decoding at the standard resolution at the step S602 inFIG. 24.

Referring to FIG. 25, there is a process for a prescribed decoding to beimplemented at the first decoder 1802 that includes an entropy decoder,an inverse quantizer, an inverse orthogonal transformer, anintra-predictor, and a motion compensator, as they are prescribed.

The first decoder 1802 first acquires a set of first sequences ofencoded bits with the standard resolution from the demultiplexer 1801,and executes a first decoding for a current region of decoding target tobe decoded (step S701), by use of the entropy decoder operating toimplement a process for a prescribed decoding to create quantized dataas decoded, the inverse quantizer operating on the quantized data toimplement a process for a prescribed inverse quantization to create dataon orthogonal transform coefficients as decoded, and the inverseorthogonal transformer operating on the data on orthogonal transformcoefficients to implement a process for a prescribed inverse orthogonaltransform to create decoded difference data.

Next, the intra-predictor operates for creation of predictive data atthe current region of decoding target, by implementing a process for aprescribed prediction based on data of a decoded region neighboring thecurrent decoding target region, to create predictive data (step S702),and makes an addition of the created predictive data and the decodeddifference data of the decoding target region, to create decoded data ofthe decoding target region.

After that, there is a determination of whether or not theintra-decoding is complete, in combination with a determination ofwhether or not a decoded picture set is obtained with a set of decodeddata created on whole regions in the planes (step S703).

Providing a repetition of the processes for any remaining encoded bitstreams to be decoded, if any intra-decoding gets complete (YES at thestep S703), then the flow goes to a step S704. If no encoded bit streamto be decoded is left, the foregoing processes for decoding come to anend.

After any intra-decoding, if this is in way to the end (NO at the stepS703), there is a subsequent decoding target region identified, beforethe flow again goes to the step S701 for the repetition of processes.

If the intra-decoding comes to the end, then the first decoder 1802enters a service for inter-decoding, where it makes a first decoding(step S704) to create decoded difference data, like the intra-decoding.

Further, the motion compensator operates to have a decoded picture setcreated as a reference picture set, acquire data on motion vectors fromthe demultiplexer 1801, and work on motion vector data using referencepictures to make a motion compensation (step S705) to create predictivepictures, and make an addition of created predictive pictures anddecoded difference data, to create a set of decoded pictures.

After that, on bases of encoded mode data or the like, there is adetermination of whether the inter-decoding has come to an end (stepS706).

If there is any subsequent decoding left as an inter-decoding (NO at thestep S706), the flow again goes to the step S704 to repeat processes forthe inter-decoding to be repeated.

If there is no subsequent decoding left as an inter-decoding (YES at thestep S706), the inter-decoding goes to the end. Here, if there comes anyencoded bit sequence to be decoded by a subsequent decoding to be anintra-decoding, the flow again goes to the step S701 to repeat theforegoing processes for decoding. If there is no encoded bit sequence tobe decoded, the processes for decoding go to an end.

Also in the ninth embodiment, there has been a moving picture decodingsystem depicted by a block diagram and described as a hardwareconfiguration, whereas like the first to the eighth embodiment, theremay well be a set of functions of the moving picture decoding system inFIG. 23 implemented as software processes using a CPU and a program.

(Tenth Embodiment)

Description is now made of an embodiment of moving picture decodingsystem according to a tenth embodiment.

FIG. 26 is a block diagram showing an example of configuration of themoving picture decoding system according to the tenth embodiment.

Referring to FIG. 26, the moving picture decoding system according tothe tenth embodiment is comprised of a set of component elements of themoving picture decoding system according to the ninth embodiment shownin FIG. 23, with the second resolution converter 1806 and the thirddecoder 1807 being omitted or kept from working, while componentelements shown in FIG. 26 are equivalent to those in FIG. 23, soredundant description is omitted.

The moving picture decoding system configured as described is adapted toacquire multiplexed sequences of encoded bits created by any of movingpicture encoding systems, moving picture encoding methods, movingpicture encoding programs, moving picture reencoding systems, movingpicture reencoding methods, and moving picture reencoding programscorresponding to this embodiment, the moving picture decoding systemaccording to this embodiment being adapted for correct decoding,affording to acquire decoded picture sequences with more diverse spatialresolutions or frequency components and higher qualities than ever, suchas decoded picture sequences with standard resolution, super-resolutionenlarged decoded picture sequences, super-resolution decoded picturesequences, and super-resolution picture sequences as decoded. It also ispossible to provide a configuration adapted to operate in accordancewith a request for a selective output of such diverse decoded picturesequences to an unshown display system.

Also in the tenth embodiment described, there has been a hardwareconfiguration depicted by a block diagram, whereas like the first to theninth embodiment, there may well be a set of functions of the movingpicture decoding system in FIG. 26 implemented as software processesusing a CPU and a program.

(Eleventh Embodiment)

Description is now made of a reencoding system according to an eleventhembodiment of the present invention.

FIG. 27 is a block diagram showing an example of configuration of thereencoding system according to the eleventh embodiment.

The reencoding system shown in FIG. 27 according to the eleventhembodiment is configured as a moving picture decoding system accordingto the tenth embodiment shown in FIG. 26, as combined with a reencoder1808 and a multiplexer 1809 additionally incorporated therein, foradaptation to a process for a re-encoding as a transcoding from acertain specific encoding style to a different encoding style else.

For instance, the reencoding system shown in FIG. 27 according to theeleventh embodiment may be configured for an AVC rewriting (or AVCtranscoding), that is, for adaptation to a process for a reencoding toan encoding style of AVC. It is noted that the moving picture decodingsystem shown in FIG. 23 according to the ninth embodiment may well beconfigured with a reencoder 1808 and a multiplexer 1809 additionallyincorporated therein.

The eleventh embodiment includes a demultiplexer 1801 adapted to providea second sequence of encoded bits with a standard resolution, which maywell be configured for a CGS (Coarse Grain Scalability) layer complyingwith MPEG-4 SVC. Further, this may well be adapted, subject to anencoding with a spatial resolution equalized between a picture sequencewith a standard resolution belonging to a base layer and asuper-resolution picture sequence with the standard resolution belongingto an extension layer referring to the base layer, to make the encodingfor creation of an encoded bit sequence complying with such a prescribedsyntax structure that meets the conditions for restriction to permit afacilitated conversion from SVC to AVC, as set forth in rare literatureson AVC rewriting (as literature files on AVC rewriting: JVT-U043,http://ftp3.itu.ch/av-arch/jvt-site/2006_10_Hangzhou/JVT-U043.zip; andJVT-V035,http://ftp3.itu.ch/av-arch/jvt-site/2007_01_Marrakech/JVT-V035.zip) inthe JVT (Joint Video Team) being a joint group of MPEG and ITU-T. Theeleventh embodiment may well be configured for services to acquire anencoded bit sequence thus created, and create an encoded bit sequenceenabling a conversion from an encoding style adapted for a handling onan extension layer configured with a CGS layer according to thisinvention, without such a reencoding as making a combination of completedecoding and encoding.

For encoded bit sequences non-adaptive to encoding services suitable tosuch the AVC rewriting, there may be a configuration to make areencoding based on an AVC encoding style for single layers, using asuper-resolution picture sequence obtained by decoding an encoded bitsequence with a standard resolution and a second encoded bit sequencewith the standard resolution, as well as data on motion vectors and dataon encoding modes of respective types or such obtained in the decodingof the encoded bit sequence with the standard resolution, for creationof an encoded bit sequence.

Accordingly, the reencoding system shown in FIG. 27 according to theeleventh embodiment may well have the component elements adapted foradditional functions as described below.

That is, there is a first decoder 1802 further adapted for functions towork on an encoded bit sequence acquired with the standard resolution,implementing processes for prescribed entropy decoding and inversequantization, to make a decoding to the state of data on orthogonaltransform coefficients at the standard resolution on a base layer,without making a complete decoding. Further, the first decoder 1802 isadapted for functions to supply the reencoder 1808 with data onorthogonal transform coefficients in way of the decoding when the AVCrewriting is determined as being possible, or with a decoded picturesequence with the standard resolution when the reencoding is determinedas being necessary. The first decoder 1802 may be adapted for functionsto supply the reencoder 1808 with data on encoding modes and data onparameters of respective types obtained in the decoding.

There is a first resolution converter 1804 further adapted for functionsto supply the reencoder 1808 with a super-resolution decoded picturesequence with the standard resolution when the reencoding is determinedas being necessary.

There is a second decoder 1805 further adapted for functions to work onan extension encoded bit sequence acquired with the standard resolution,implementing processes for prescribed entropy decoding and inversequantization, to make a decoding to the state of data on orthogonaltransform coefficients at the standard resolution on an extension layer,without making a complete decoding, as well as for functions to supplythe reencoder 1808 with data on orthogonal transform coefficients at thestandard resolution on the extension layer in way of the decoding whenthe AVC rewriting is determined as being possible, or with asuper-resolution picture sequence as decoded with the standardresolution when the reencoding is determined as being necessary.

The reencoder 1808 is adapted for functions to work when the AVCrewriting is determined as being possible, to acquire from the firstdecoder 1802 data on orthogonal transform coefficients in way of thedecoding, and from the second decoder 1805 data on orthogonal transformcoefficients at the standard resolution on the extension layer in way ofthe decoding, make a synthesis of respective data on orthogonaltransform coefficients, and afterward implement a process for aprescribed entropy decoding, to create an encoded bit sequence asreencoded, and supply to the multiplexer 1809. The reencoder 1808 isadapted for functions to work when the reencoding is determined as beingnecessary, to acquire from the first decoder 1802 a decoded picturesequence with the standard resolution, from the first resolutionconverter 1804 a super-resolution decoded picture sequence with thestandard resolution, and from the second decoder 1805 a super-resolutionpicture sequence as decoded with the standard resolution, as necessary,to implement a process for a prescribed encoding, to create an encodedbit sequence as reencoded, and supply to the multiplexer 1809.

The multiplexer 1809 is adapted for functions to acquire an encoded bitsequence as reencoded and supplied, and work complying with a prescribedsyntax structure, to implement a process for a multiplexing, whileinserting identification data for identification of sets of data used inthe encoding, involving data on encoding modes and data on parameters ofrespective types, to create an encoded bit sequence as multiplexed tooutput.

The moving picture reencoding system configured as described is adaptedto acquire multiplexed sequences of encoded bits created at a movingpicture encoding system according to any of the first to the eighthembodiment or the like, in addition to adaptation of the moving picturereencoding system according to this invention to acquire from the firstdecoder 1802 data on orthogonal transform coefficients in way of thedecoding and from the second decoder 1805 data on orthogonal transformcoefficients in way of the decoding at the standard resolution on anextension layer, make a synthesis of respective data on orthogonaltransform coefficients, and again implement a process for entropyencoding and a process for a multiplexing, to create an encoded bitsequence as reencoded, affording in a facilitated manner to reencodedecoded components with more diverse spatial resolutions or frequencycomponents and higher qualities than ever, without making a fulldecoding for creation of decoded picture sequences, and withoutconsuming many operations for calculation.

Further, it is possible to provide a configuration adapted to create anencoded bit sequence as reencoded for conversion into a differentencoding style, permitting creation of an encoded bit sequence adaptiveto a targeted encoding style.

It is possible to provide decoded picture sequences obtained with morediverse spatial resolutions or frequency components and higher qualitiesthan ever, such as a decoded picture sequence with standard resolution,a super-resolution decoded picture sequence, and a super-resolutionpicture sequence as decoded. Further, it is possible to make areencoding on such various decoded picture sequences, to create encodedbit sequences on a single layer.

Accordingly, for instance, in application to those devices such asmobile terminal display devices having a moving picture decoding devicemounted thereon under restrictions such as to calculation rate or powerconsumption, and simply adaptive to such a decoding as enabled with thelower calculation rate or power consumption, or lower encodingtransmission rate, the moving picture reencoding system according tothis invention can be configured for a reencoding to permit creation ofan encoded bit sequence adaptive to a targeted encoding style.

The moving picture reencoding system configured as described is adaptedto acquire multiplexed sequences of encoded bits created at a movingpicture encoding system according to any of the first to the eighthembodiment or the like, in addition to adaptation of the moving picturereencoding system according to this invention to make a correctdecoding, affording to obtain decoded picture sequences with morediverse spatial resolutions or frequency components and higher qualitiesthan ever, such as decoded picture sequences with standard resolution,super-resolution decoded picture sequences, and super-resolution picturesequences as decoded.

Further, it is possible to make a reencoding on such various decodedpicture sequences, to create encoded bit sequences on a single layer.Therefore, for instance, in application to those devices such as mobileterminal display devices having a moving picture decoding device mountedthereon under restrictions such as to calculation rate or powerconsumption, and simply adaptive to such a decoding as enabled with thelower calculation rate or power consumption, or lower encodingtransmission rate, the moving picture reencoding system according tothis invention can be configured to make a reencoding for conversion toa different encoding style, permitting creation of an encoded bitsequence adaptive to a targeted encoding style.

Description is now made of actions of the reencoding system accordingshown in FIG. 27, with reference to a flowchart of FIG. 28.

First, the demultiplexer 1801 acquires a sequence of multiplexed bitsinput thereto, and works thereon complying with a prescribed syntaxstructure, to make a demultiplexing (step S701) while identifyingidentification data for identification of among others data on encodingmodes and data on parameters of respective types used in encodingprocesses, on one hand to thereby obtain from the sequence ofmultiplexed bits a first sequence of encoded bits with the standardresolution and a second sequence of encoded bits with the standardresolution, and on the other hand to further obtain, if available fromthe sequence of multiplexed bits, a sequence of bits of data onenlargement ratio, a sequence of bits of data on resolution conversionratio, and a sequence of encoded bits of data on an extension predictionat the standard resolution. After that, the demultiplexer 1801 works tosupply the first decoder 1802 with the first sequence of encoded bitsand the second decoder 1805 with second sequence of encoded bits, asthey are obtained, respectively, and further to obtain data on motionvectors, if available from the sequence of multiplexed bits, and supplyto the first decoder 1802.

After that, the demultiplexer 1801 determines whether the AVC rewritingis possible or impossible (step S702). This determination may be basedon determination data contained in the sequence of multiplexed bits, foruse to determine whether or not the AVC rewriting is possible.

In this system, if the AVC rewriting is possible (YES at the step S702),then the flow branches into a first parallel process (steps S703) and asecond parallel process (step S704), to enter subsequent parallelactions.

To the contrary, in this system, if the AVC rewriting is impossible (NOat the step S702), then determining a reencoding as being necessary, theflow branches into a third parallel process (step S707 and step S708)and a fourth parallel process (step S709), to enter subsequent parallelactions.

If the AVC rewriting is possible, the flow goes to the followingprocesses.

In the first parallel process, the first decoder 1802 acquires the firstsequence of encoded bits, and implements thereon a process for aprescribed entropy decoding (step S703) combined with a process for aninverse quantization, to create data on orthogonal transformcoefficients on the base layer in way of the decoding, and supply to thereencoder 1808. After that, it waits for completion of the secondparallel process.

In the second parallel process, the second decoder 1805 acquires thesecond sequence of encoded bits, and implements thereon a process for aprescribed entropy decoding (step S704) combined with a process for aninverse quantization, to create data on orthogonal transformcoefficients on the extension layer in way of the decoding, and supplyto the reencoder 1808. After that, it waits for completion of the firstparallel process.

With the first and the second parallel process completed, the reencoder1808 acquires from the first decoder 1802 data on orthogonal transformcoefficients on the base layer in way of the decoding, and from thesecond decoder 1805 data on orthogonal transform coefficients on theextension layer in way of the decoding, and makes thereon a synthesisbased on a process for a prescribed synthesizing (step S705), making aprescribed entropy encoding on a result of the synthesis (step S706), tocreate a sequence of encoded bits as reencoded, and supply to themultiplexer 1809. The foregoing processes are processes when the AVCrewriting is possible.

If the AVC rewriting is impossible, that is when the reencoding isnecessary, the flow goes to the following processes.

In the third parallel process, the first decoder 1802 works to acquirefrom the demultiplexer 1801 a sequence of encoded bits with the standardresolution, and acquire, if available, data on motion vectors, andimplements thereon a process for a decoding at the standard resolution(step S707), to create a sequence of decoded pictures, and supply to thesecond decoder 1805, and to the reencoder 1808 as necessary.

Then, the second decoder 1805 acquires the sequence of decoded picturescreated with the standard resolution at the first decoder 1802, andstores in a prescribed buffer for temporary accumulation (step S708).After that, it waits for completion of the fourth parallel process.

In the fourth parallel process, the second decoder 1805 acquires thesecond sequence of encoded bits from the demultiplexer 1801, andimplements thereon a process for a prescribed decoding (step S709), tocreate decoded difference data with the standard resolution. After that,it waits for completion of the third parallel process.

With the third and the fourth parallel process completed, a firstsuper-resolution enlarger 1803 acquires the sequence of decoded pictureswith the standard resolution from the first decoder 1802, and implementsthereon a process for a prescribed super-resolution enlargement (stepS710), to create a sequence of super-resolution enlarged decodedpictures with a high resolution being a resolution higher than thestandard resolution, and supply to the first resolution converter 1804.

Then, the first resolution converter 1804 acquires the sequence ofsuper-resolution enlarged decoded pictures with the high resolution fromthe first super-resolution enlarger 1803, and implements thereon aprocess for a prescribed resolution conversion (step S711) to create,from the sequence of super-resolution enlarged decoded pictures with thehigh resolution being a resolution higher than the standard resolution,a sequence of super-resolution decoded pictures with the standardresolution, and supply to the second decoder 1805, and to the reencoder1808 as necessary.

Then, the second decoder 1805 acquires the sequence of super-resolutiondecoded pictures created with the standard resolution at the firstresolution converter 1804, and stores in a prescribed buffer fortemporary accumulation (step S712).

The second decoder 1805 implements a process for a prescribed prediction(step S713) to create predictive pictures with the standard resolution,and makes an addition of the predictive pictures created with thestandard resolution and decoded difference data at the standardresolution, to create a sequence of super-resolution pictures as decodedwith the standard resolution, and supply to the reencoder 1808 asnecessary.

The reencoder 1808 thus acquires the sequence of decoded pictures withthe standard resolution from the first decoder 1802, the sequence ofsuper-resolution decoded pictures with the standard resolution from thefirst resolution converter 1804, and the sequence of super-resolutionpictures with the standard resolution from the second decoder 1805, asnecessary, and implements thereon a process for a prescribed encoding tothereby make a reencoding (step S714) to create a sequence of encodedbits, and supply to the multiplexer 1809.

The multiplexer 1809 acquires sequences of encoded bit as reencoded andsupplied, and works complying with a prescribed syntax structure, toimplement a process for a multiplexing, while inserting identificationdata for identification of sets of data used in the encoding, involvingdata on encoding modes and data on parameters of respective types, tocreate a sequence of encoded bit as multiplexed to output (step S715).The present embodiment involves a series of actions to be completethrough the foregoing steps.

According to the present embodiment, there is a moving picturereencoding system operable by execution of the foregoing steps,permitting creation of a sequence of encoded bits as reencoded. It isnoted that there have been operations described as being parallelprocesses according to the present embodiment, while those processed inparallel may be consecutively processed in a configuration operableaccording to the present embodiment.

Also in the eleventh embodiment, there has been a configuration depictedby a block diagram and described as a hardware, whereas like the firstto the tenth embodiment, there may well be a set of functions of thereencoding system in FIG. 27 implemented as software processes using aCPU and a program.

According to the present invention, there are moving picture decodingsystems, moving picture decoding methods, and moving picture decodingprograms, as well as moving picture reencoding systems, moving picturereencoding methods, and moving picture reencoding programs, having theirranges of application to such apparatuses or systems, methods, andprograms as adapted for encoding and decoding moving pictures, withoutspecific limitations. For the present invention, applications cited mayinvolve, for instance, broadcasting equipment typified by TV, mobiletelephones, teleconferences, monitors, reproducing systems using arecording medium such as CD, DVD, or Blue-Ray Disc®, recording andreproducing systems using a recordable and rewritable recording mediumsuch as DVD-R/RW or BD-R/RW, HDD, or SD, imaging, recording andreproducing systems such as a camcorder, recording and editing systemssuch as a an authoring system, and delivering systems for movingpictures.

INDUSTRIAL APPLICABILITY

According to the present invention, there are systems such as a movingpicture encoding system, having a first encoder working for services toencode moving pictures input with a standard resolution, in combinationwith a first super-resolution enlarger working on moving pictures inputwith the standard resolution, implementing a process for asuper-resolution enlargement including information on frequencycomponents in the spatial direction and the temporal direction that hasbeen potentially contained in the input moving pictures but unable toexpress to a sufficient degree by the standard resolution, followed byimplementing processes such as for a prescribed resolution conversion ata first resolution converter, thus permitting a second encoder to makean encoding of moving pictures based on an increased amount ofinformation relative to an information amount of moving pictures inputwith the standard resolution, as a resultant effect. At the secondencoder, as reference signals for the encoding there may be use of,among others, decoded signals as decoded at the first encoder or asadditionally processed through processes such as for a secondsuper-resolution enlargement and a second resolution conversion.

Moreover, according to the present invention, there are systems such asa moving picture encoding system, having a first encoder working forservices to encode moving pictures input with a standard resolution, incombination with a first super-resolution enlarger working on movingpictures input with the standard resolution, implementing a process fora super-resolution enlargement including information on frequencycomponents in the spatial direction and the temporal direction that hasbeen potentially contained in the input moving pictures but unable toexpress to a sufficient degree by the standard resolution, followed byan encoding of thus obtained signals at a third encoder, permitting thethird encoder to make an encoding of moving pictures based on anincreased amount of information relative to an information amount ofmoving pictures input with the standard resolution, as a resultanteffect. At the third encoder, as reference signals for the encodingthere may be use of decoded signals as decoded at the first encoder andprocessed through processes such as for a second super-resolutionenlargement or a third resolution conversion to enhance the resolutionup to the same resolution as first super-resolution enlarged signals.Still more, according to the present invention, there are systems suchas a moving picture decoding system, adapted to input thus encodedmoving pictures to decode, and yet more, according to the presentinvention, there are systems such as a moving picture reencoding system,adapted to input thus encoded moving pictures to decode and reencode.

REFERENCE SIGNS LIST

-   101 first accumulation buffer-   102 first encoder-   103 first super-resolution enlarger-   104 first resolution converter-   105 second super-resolution enlarger-   106 second resolution converter-   107 second encoder-   108 third encoder-   109 multiplexer-   110 third resolution converter-   1801 demultiplexer-   1802 first decoder-   1803 first super-resolution enlarger-   1804 first resolution converter-   1805 second decoder-   1806 second resolution converter-   1807 third decoder-   1808 reencoder-   1809 multiplexer

The invention claimed is:
 1. A moving picture encoding system comprising: a first encoder configured to work on a subsequence of a sequence of moving pictures with a standard resolution to implement a first combination of processes for an encoding and a decoding to create a first sequence of encoded bits and a set of decoded pictures with the standard resolution; a first super-resolution enlarger configured to work on the subsequence of the sequence of moving pictures with the standard resolution to implement an interpolation of pixels with a first enlargement to create a set of super-resolution enlarged pictures with a first resolution higher than the standard resolution; a first resolution converter configured to work on the set of super-resolution enlarged pictures to implement a process for a first resolution conversion to create a set of super-resolution enlarged and converted pictures with a standard resolution; a second super-resolution enlarger configured to acquire the set of decoded pictures with the standard resolution from the first encoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution; a second resolution converter configured to work on the set of super-resolution enlarged decoded pictures to implement a process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and a second encoder configured to: have the set of super-resolution enlarged and converted pictures from the first resolution converter as a set of encoding target pictures, the set of decoded pictures from the first encoder as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures from the second resolution converter as a set of second reference pictures, select one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and implement a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information, wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 2. A moving picture decoding system comprising: a demultiplexer configured to work on a sequence of input encoded bits to implement a process for a prescribed demultiplexing to output at least a first and a second sequence of encoded bits; a first decoder configured to acquire the first sequence of encoded bits obtained with a standard resolution at the demultiplexer to implement thereon a process for a prescribed first decoding to create a sequence of decoded pictures with a standard resolution; a first super-resolution enlarger configured to acquire the sequence of decoded pictures created with a standard resolution at the first decoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement to create a sequence of super-resolution enlarged decoded pictures with a first resolution higher than a standard resolution; a first resolution converter configured to acquire the sequence of super-resolution enlarged decoded pictures created at the first super-resolution enlarger to work on the sequence of super-resolution enlarged decoded pictures to implement a process for a prescribed resolution conversion to create a sequence of super-resolution decoded pictures with a standard resolution; a second decoder configured to acquire the second sequence of encoded bits obtained with a standard resolution at the demultiplexer as a set of decoding targets, the sequence of decoded pictures created with the standard resolution at the first decoder as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution at the first resolution converter as a set of second reference pictures, and select one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, to create a sequence of super-resolution pictures decoded with the standard resolution based on the set of decoding targets and the set of selected reference pictures; and a second resolution converter configured to acquire the sequence of decoded pictures with the standard resolution from the first decoder to work on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement to create a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution, wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 3. A moving picture decoding method comprising: a step of implementing a process for a prescribed demultiplexing on a sequence of input encoded bits, outputting at least a first and a second sequence of encoded bits; a step of acquiring the first sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing, implementing thereon a process for a prescribed first decoding, creating a sequence of decoded pictures with the standard resolution; a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement, creating a sequence of super-resolution enlarged decoded pictures with a first resolution higher than the standard resolution; a step of acquiring the sequence of super-resolution enlarged decoded pictures created through the process for the prescribed super-resolution enlargement working on the sequence of super-resolution enlarged decoded pictures to implement a process for a first resolution conversion, creating a sequence of super-resolution decoded pictures with a standard resolution; a step of acquiring the second sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing as a set of decoding targets, the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution through the process for the prescribed resolution conversion as a set of second reference pictures, and selecting one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, based on the set of decoding targets and the set of selected reference pictures to create a sequence of super-resolution pictures decoded with the standard resolution; and a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding working on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement, creating a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution, wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 4. A recording medium storing a moving picture decoding program comprising a non-transitory computer-readable medium configured to have a computer execute: a step of implementing a process for a prescribed demultiplexing on a sequence of input encoded bits, outputting at least a first and a second sequence of encoded bits; a step of acquiring the first sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing, implementing thereon a process for a prescribed first decoding, creating a sequence of decoded pictures with the standard resolution; a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement, creating a sequence of super-resolution enlarged decoded pictures with a first resolution higher than the standard resolution; a step of acquiring the sequence of super-resolution enlarged decoded pictures created through the process for the prescribed super-resolution enlargement working on the sequence of super-resolution enlarged decoded pictures to implement a process for a first resolution conversion, creating a sequence of super-resolution decoded pictures with a standard resolution; a step of acquiring the second sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing as a set of decoding targets, the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution through the process for the prescribed resolution conversion as a set of second reference pictures, and selecting one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, based on the set of decoding targets and the set of selected reference pictures to create a sequence of super-resolution pictures decoded with the standard resolution; and a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement creating a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution, wherein the first resolution is different from the second resolution, wherein the standard resolution of the second sequence of encoded bits is the same as the standard resolution of the super-resolution decoded pictures, and wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 5. The moving picture encoding system according to claim 1, wherein the first super-resolution enlarger comprises: a positioner configured to work in a processing for super-resolution enlargement, on a combination of a base picture constituting a base therein and one or more observation pictures to be based on, to make a positioning to pixel positions of a desirable high resolution, for an interval-unequal sampling to create nonhomogeneous high resolution pictures; an interpolator configured to work on nonhomogeneous high resolution pictures created at the positioner, to implement a process for a prescribed nonhomogeneous interpolation to create interpolated pictures with a desirable high resolution; an estimated picture creator configured to acquire interpolated pictures created at the interpolator, to implement thereon a process for a prescribed reconstruction to create estimated pictures with a desirable resolution; and a repetition determiner configured to acquire a nonhomogeneous high resolution picture from the positioner and a nonhomogeneous estimated picture from the estimated picture creator, for employment of the acquired pictures to follow a prescribed determination method to determine whether or not a repetition of the processing for super-resolution enlargement is necessary, and work depending on a result thereof, to operate as the repetition is necessary, to provide the interpolator with information for a control to continue the processing, and operate as the repetition is unnecessary, to provide the estimated picture creator with information for a control to output a set of estimated pictures with a high resolution after super-resolution enlargement.
 6. The moving picture encoding system according to claim 1, further comprising a multiplexer configured to operate complying with a prescribed syntax structure to multiplex the first sequence of encoded bits from the first encoder, the second sequence of encoded bits from the second encoder, and information on encoding parameters of respective types used in the encoding at the first encoder and the encoding at the second encoder.
 7. The moving picture encoding system according to claim 1, further comprising: a first accumulation buffer configured to work in a course before the first encoder and the first super-resolution enlarger, to accumulate subsequences of the sequence of moving pictures with the standard resolution; a second accumulation buffer configured to work in a course between the first encoder and the second encoder, to accumulate sets of decoded pictures from the first encoder; an accumulation controller configured to work for detections of buffer accumulation amounts at the first accumulation buffer, the second accumulation buffer, the first encoder, and the second encoder, to control the buffer accumulation amounts; and a code rate controller configured to work on bases of the detections of buffer accumulation amounts at the accumulation controller, to control code rates at the first encoder and the second encoder.
 8. A moving picture encoding method comprising: a step of implementing a first combination of processes for an encoding and a decoding on a sequence of moving pictures with a standard resolution, creating a first sequence of encoded bits and a set of decoded pictures with the standard resolution; a step of implementing an interpolation of pixels with a first enlargement on the sequence of moving pictures with the standard resolution, creating a set of super-resolution enlarged pictures with a first resolution higher than a standard resolution; a step of implementing a process for a first resolution conversion on the set of super-resolution enlarged pictures, creating a set of super-resolution enlarged and converted pictures with a standard resolution; a step of acquiring the set of decoded pictures with the standard resolution to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution; a step of working on the set of super-resolution enlarged decoded pictures to implement process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and a step of: having the set of super-resolution enlarged and converted pictures as a set of encoding target pictures, the set of decoded pictures as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures as a set of second reference pictures, selecting one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and implementing a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information, wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 9. A recording medium storing a moving picture encoding program comprising a non-transitory computer-readable medium configured to have a computer execute: a step of implementing a first combination of processes for an encoding and a decoding on a sequence of moving pictures with a standard resolution, creating a first sequence of encoded bits and a set of decoded pictures with the standard resolution; a step of implementing an interpolation of pixels with a first enlargement on the sequence of moving pictures with the standard resolution, creating a set of super-resolution enlarged pictures with a first resolution higher than the standard resolution; a step of implementing process for a first resolution conversion on the set of super-resolution enlarged pictures, creating a set of super-resolution enlarged and converted pictures with a standard resolution; a step of acquiring the set of decoded pictures with the standard resolution from the first encoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution; a step of working on the set of super-resolution enlarged decoded pictures to implement a process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and a step of: having the set of super-resolution enlarged and converted pictures as a set of encoding target pictures, the set of decoded pictures as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures as a set of second reference pictures, selecting one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and implementing a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information, wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.
 10. The moving picture encoding system according to claim 1, wherein the second encoder divides an encoding target picture into a plurality of regions to select one of the set of first reference pictures and the set of second reference pictures for each divided region.
 11. The moving picture decoding system according to claim 2, wherein the second decoder divides a decoding target into a plurality of regions to select one of the set of first reference pictures and the set of second reference pictures for each divided region. 